fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct — the timeout covered the WHOLE archive, not per object. Importer._import_archive (importer.py:409) runs the full per-member pipeline (sha256 + pHash + dedup query + copy + provenance) for EVERY media member inline, all under import_media_file's single 300s soft limit. A single media file is sub-second; a multi-hundred-member archive blows the budget. They shared one task name and one timeout. **Split archive into its own task** - New `import_archive_file` task: same body as import_media_file (dispatch is by file-kind inside Importer.import_one) but soft=30min / hard=35min. Shared `_run_import_task` helper holds the flip-to-processing + resilience-contract wrapper; both tasks call it. - New `enqueue_import(task_id, task_type)` router — single source of truth for media-vs-archive dispatch. Used by all three enqueue sites: scan_directory, /api/import/retry-failed, recover_interrupted_tasks. - scan_directory now sets ImportTask.task_type = "archive" when is_archive(entry) (the model field already existed, anticipating this; scan was hardcoding "media"). - import_archive_file routes to the existing 'import' queue via the task_routes `import_file.*` wildcard — no worker config change. **Archive-aware recovery sweeps** Both sweeps would otherwise preempt a legitimately-running archive: - recover_interrupted_tasks (ImportTask 'processing' sweep): now task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the 35-min hard limit). Single UPDATE with an OR predicate over the two (task_type, cutoff) pairs; requeue routes via enqueue_import. - recover_stalled_task_runs (TaskRun 'running' sweep): now supports per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above the per-queue overrides added for ml. import_archive_file gets 40 min while the 'import' queue stays at the 5-min default for single-file imports. Precedence: task_name → queue → default, each pass excluding rows claimed by a higher-precedence pass so every row is touched once. **Tests** - test_import_archive_file_registered - test_recover_stalled_task_runs_archive_task_uses_longer_threshold — pins that a 10-min archive task-run survives, a 50-min one is flagged, and a same-queue 10-min media import is flagged at the default. - _make_task_run gains queue= + task_name= params. After deploy: archive imports get a 30-min budget and aren't preempted by either sweep; single-file imports keep their tight 5-min detection.
This commit is contained in:
@@ -195,12 +195,13 @@ def test_cleanup_old_deletes_finished_old(db_sync):
|
||||
|
||||
|
||||
def _make_task_run(db_sync, *, status, started_at, finished_at=None,
|
||||
error_type=None, queue="default"):
|
||||
error_type=None, queue="default",
|
||||
task_name="backend.app.tasks.fake.t"):
|
||||
from backend.app.models import TaskRun
|
||||
row = TaskRun(
|
||||
celery_task_id="x",
|
||||
queue=queue,
|
||||
task_name="backend.app.tasks.fake.t",
|
||||
task_name=task_name,
|
||||
target_id=1,
|
||||
started_at=started_at,
|
||||
finished_at=finished_at,
|
||||
@@ -296,6 +297,53 @@ def test_recover_stalled_task_runs_ml_queue_uses_longer_threshold(db_sync):
|
||||
assert ml_fresh_status == "running"
|
||||
assert ml_stale_status == "error"
|
||||
|
||||
|
||||
def test_recover_stalled_task_runs_archive_task_uses_longer_threshold(db_sync):
|
||||
"""import_archive_file shares the 'import' queue with fast
|
||||
single-file import_media_file, so it gets a per-task-name override
|
||||
(40 min) while the import queue stays at the 5-min default. A
|
||||
10-min-old archive task-run must survive; a 50-min-old one is
|
||||
flagged. Operator-flagged 2026-05-28."""
|
||||
from sqlalchemy import select
|
||||
|
||||
from backend.app.models import TaskRun
|
||||
from backend.app.tasks.maintenance import recover_stalled_task_runs
|
||||
|
||||
archive_name = "backend.app.tasks.import_file.import_archive_file"
|
||||
now = datetime.now(UTC)
|
||||
# Fast single-file import on the same queue, 10 min old → flagged
|
||||
# by the default 5-min rule.
|
||||
media_id = _make_task_run(
|
||||
db_sync, status="running", queue="import",
|
||||
task_name="backend.app.tasks.import_file.import_media_file",
|
||||
started_at=now - timedelta(minutes=10),
|
||||
)
|
||||
# Archive on the same queue, 10 min old → survives (40-min override).
|
||||
archive_fresh_id = _make_task_run(
|
||||
db_sync, status="running", queue="import",
|
||||
task_name=archive_name,
|
||||
started_at=now - timedelta(minutes=10),
|
||||
)
|
||||
# Archive 50 min old → past even the 40-min override → flagged.
|
||||
archive_stale_id = _make_task_run(
|
||||
db_sync, status="running", queue="import",
|
||||
task_name=archive_name,
|
||||
started_at=now - timedelta(minutes=50),
|
||||
)
|
||||
db_sync.commit()
|
||||
|
||||
recovered = recover_stalled_task_runs.apply().get()
|
||||
assert recovered == 2 # media + stale archive
|
||||
|
||||
db_sync.expire_all()
|
||||
def _status(_id):
|
||||
return db_sync.execute(
|
||||
select(TaskRun.status).where(TaskRun.id == _id)
|
||||
).scalar_one()
|
||||
assert _status(media_id) == "error"
|
||||
assert _status(archive_fresh_id) == "running"
|
||||
assert _status(archive_stale_id) == "error"
|
||||
|
||||
db_sync.expire_all()
|
||||
status = db_sync.execute(
|
||||
select(TaskRun.status).where(TaskRun.id == fresh_id)
|
||||
|
||||
Reference in New Issue
Block a user