fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.

**Split archive into its own task**

- New `import_archive_file` task: same body as import_media_file
  (dispatch is by file-kind inside Importer.import_one) but
  soft=30min / hard=35min. Shared `_run_import_task` helper holds the
  flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
  truth for media-vs-archive dispatch. Used by all three enqueue sites:
  scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
  is_archive(entry) (the model field already existed, anticipating
  this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
  task_routes `import_file.*` wildcard — no worker config change.

**Archive-aware recovery sweeps**

Both sweeps would otherwise preempt a legitimately-running archive:

- recover_interrupted_tasks (ImportTask 'processing' sweep): now
  task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
  get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
  35-min hard limit). Single UPDATE with an OR predicate over the two
  (task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
  per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
  the per-queue overrides added for ml. import_archive_file gets 40 min
  while the 'import' queue stays at the 5-min default for single-file
  imports. Precedence: task_name → queue → default, each pass excluding
  rows claimed by a higher-precedence pass so every row is touched once.

**Tests**

- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
  pins that a 10-min archive task-run survives, a 50-min one is flagged,
  and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.

After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
This commit is contained in:
2026-05-27 22:45:11 -04:00
parent 407de18ff6
commit a85880f965
6 changed files with 232 additions and 101 deletions
+50 -2
View File
@@ -195,12 +195,13 @@ def test_cleanup_old_deletes_finished_old(db_sync):
def _make_task_run(db_sync, *, status, started_at, finished_at=None,
error_type=None, queue="default"):
error_type=None, queue="default",
task_name="backend.app.tasks.fake.t"):
from backend.app.models import TaskRun
row = TaskRun(
celery_task_id="x",
queue=queue,
task_name="backend.app.tasks.fake.t",
task_name=task_name,
target_id=1,
started_at=started_at,
finished_at=finished_at,
@@ -296,6 +297,53 @@ def test_recover_stalled_task_runs_ml_queue_uses_longer_threshold(db_sync):
assert ml_fresh_status == "running"
assert ml_stale_status == "error"
def test_recover_stalled_task_runs_archive_task_uses_longer_threshold(db_sync):
"""import_archive_file shares the 'import' queue with fast
single-file import_media_file, so it gets a per-task-name override
(40 min) while the import queue stays at the 5-min default. A
10-min-old archive task-run must survive; a 50-min-old one is
flagged. Operator-flagged 2026-05-28."""
from sqlalchemy import select
from backend.app.models import TaskRun
from backend.app.tasks.maintenance import recover_stalled_task_runs
archive_name = "backend.app.tasks.import_file.import_archive_file"
now = datetime.now(UTC)
# Fast single-file import on the same queue, 10 min old → flagged
# by the default 5-min rule.
media_id = _make_task_run(
db_sync, status="running", queue="import",
task_name="backend.app.tasks.import_file.import_media_file",
started_at=now - timedelta(minutes=10),
)
# Archive on the same queue, 10 min old → survives (40-min override).
archive_fresh_id = _make_task_run(
db_sync, status="running", queue="import",
task_name=archive_name,
started_at=now - timedelta(minutes=10),
)
# Archive 50 min old → past even the 40-min override → flagged.
archive_stale_id = _make_task_run(
db_sync, status="running", queue="import",
task_name=archive_name,
started_at=now - timedelta(minutes=50),
)
db_sync.commit()
recovered = recover_stalled_task_runs.apply().get()
assert recovered == 2 # media + stale archive
db_sync.expire_all()
def _status(_id):
return db_sync.execute(
select(TaskRun.status).where(TaskRun.id == _id)
).scalar_one()
assert _status(media_id) == "error"
assert _status(archive_fresh_id) == "running"
assert _status(archive_stale_id) == "error"
db_sync.expire_all()
status = db_sync.execute(
select(TaskRun.status).where(TaskRun.id == fresh_id)