bvandeusen
a85880f965
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct — the timeout covered the WHOLE archive, not per object. Importer._import_archive (importer.py:409) runs the full per-member pipeline (sha256 + pHash + dedup query + copy + provenance) for EVERY media member inline, all under import_media_file's single 300s soft limit. A single media file is sub-second; a multi-hundred-member archive blows the budget. They shared one task name and one timeout. **Split archive into its own task** - New `import_archive_file` task: same body as import_media_file (dispatch is by file-kind inside Importer.import_one) but soft=30min / hard=35min. Shared `_run_import_task` helper holds the flip-to-processing + resilience-contract wrapper; both tasks call it. - New `enqueue_import(task_id, task_type)` router — single source of truth for media-vs-archive dispatch. Used by all three enqueue sites: scan_directory, /api/import/retry-failed, recover_interrupted_tasks. - scan_directory now sets ImportTask.task_type = "archive" when is_archive(entry) (the model field already existed, anticipating this; scan was hardcoding "media"). - import_archive_file routes to the existing 'import' queue via the task_routes `import_file.*` wildcard — no worker config change. **Archive-aware recovery sweeps** Both sweeps would otherwise preempt a legitimately-running archive: - recover_interrupted_tasks (ImportTask 'processing' sweep): now task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the 35-min hard limit). Single UPDATE with an OR predicate over the two (task_type, cutoff) pairs; requeue routes via enqueue_import. - recover_stalled_task_runs (TaskRun 'running' sweep): now supports per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above the per-queue overrides added for ml. import_archive_file gets 40 min while the 'import' queue stays at the 5-min default for single-file imports. Precedence: task_name → queue → default, each pass excluding rows claimed by a higher-precedence pass so every row is touched once. **Tests** - test_import_archive_file_registered - test_recover_stalled_task_runs_archive_task_uses_longer_threshold — pins that a 10-min archive task-run survives, a 50-min one is flagged, and a same-queue 10-min media import is flagged at the default. - _make_task_run gains queue= + task_name= params. After deploy: archive imports get a 30-min budget and aren't preempted by either sweep; single-file imports keep their tight 5-min detection.
34 lines
1.2 KiB
Python
34 lines
1.2 KiB
Python
"""Confirms every FC-2a task module imports cleanly and registers with Celery.
|
|
|
|
Celery's `include=[...]` parameter on the Celery() constructor is lazy — it
|
|
only imports those modules when a worker boots. To assert task registration
|
|
in a plain pytest run we explicitly import the task modules ourselves.
|
|
"""
|
|
|
|
# Importing for side-effect: each module's @celery.task decorators register
|
|
# the task with the global Celery instance at module import time.
|
|
import backend.app.tasks.import_file # noqa: F401
|
|
import backend.app.tasks.scan # noqa: F401
|
|
import backend.app.tasks.thumbnail # noqa: F401
|
|
from backend.app.celery_app import celery
|
|
|
|
|
|
def test_scan_task_registered():
|
|
assert "backend.app.tasks.scan.scan_directory" in celery.tasks
|
|
|
|
|
|
def test_import_media_file_registered():
|
|
assert "backend.app.tasks.import_file.import_media_file" in celery.tasks
|
|
|
|
|
|
def test_import_archive_file_registered():
|
|
assert "backend.app.tasks.import_file.import_archive_file" in celery.tasks
|
|
|
|
|
|
def test_generate_thumbnail_registered():
|
|
assert "backend.app.tasks.thumbnail.generate_thumbnail" in celery.tasks
|
|
|
|
|
|
def test_backfill_thumbnails_registered():
|
|
assert "backend.app.tasks.thumbnail.backfill_thumbnails" in celery.tasks
|