feat(import-resilience L2): one-shot re-download for corrupt downloaded files
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its source, bounded to a single attempt. Operator-requested 2026-05-28. New backend/app/services/refetch_service.py: - resolve_refetch_source: parse the failed file's sidecar → platform, derive the artist from the import path, find an ENABLED Source with a real feed URL for (artist, platform). Returns None for filesystem-only imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic anchors (not pollable). - attempt_refetch: if not already refetched AND a Source resolves, delete the corrupt file (so gallery-dl's skip_existing re-fetches it), set ImportTask.refetched=True, and trigger ONE download_source re-check. Bounded by `refetched` so source-side corruption can't loop. Wiring: - Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed' tasks). Returns refetch_queued / no_source / already_refetched / not_found / not_failed. - Auto path in recover_interrupted_tasks: for each poison-pill row, if env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the manual button is the primary path; auto is opt-in since re-fetch deletes a file + re-runs the downloader). - Frontend: a cloud-refresh icon button on failed rows in ImportTaskList → stores.import.refetchTask → toast keyed on the result status. Filesystem imports with no upstream return no_source — the operator's only remediation there is replacing the file on disk, surfaced clearly in the toast. Tests: 404 unknown task, 400 non-failed task, no_source when unresolvable, and the full resolvable-source path (file deleted, refetched flag set, one download_source dispatched, second call is a no-op). The resolvable test repoints the migration-seeded import_settings(id=1) scan path rather than inserting a conflicting row.
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
"""Periodic maintenance: recover stuck import tasks, garbage-collect old finished tasks."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
@@ -183,6 +184,28 @@ def recover_interrupted_tasks() -> int:
|
||||
for tid, task_type in stuck:
|
||||
enqueue_import(tid, task_type)
|
||||
|
||||
# Layer-2 auto re-download (env-gated, default OFF). For each
|
||||
# poison-pill row that resolves to a pollable Source, delete the
|
||||
# bad file and trigger ONE source re-check to fetch a fresh
|
||||
# copy. Bounded by ImportTask.refetched so source-side
|
||||
# corruption can't loop. The 'failed' row stays as history; the
|
||||
# re-downloaded file re-imports as a fresh task on the next scan.
|
||||
if poison_ids and os.environ.get("FC_AUTO_REFETCH_CORRUPT", "0") == "1":
|
||||
from ..models import ImportSettings
|
||||
from ..services.refetch_service import attempt_refetch
|
||||
import_root = Path(session.execute(
|
||||
select(ImportSettings.import_scan_path)
|
||||
.where(ImportSettings.id == 1)
|
||||
).scalar_one())
|
||||
for pid in poison_ids:
|
||||
ptask = session.get(ImportTask, pid)
|
||||
if ptask is None:
|
||||
continue
|
||||
try:
|
||||
attempt_refetch(session, ptask, import_root)
|
||||
except Exception as exc: # noqa: BLE001 — best-effort
|
||||
log.warning("auto-refetch failed for task %s: %s", pid, exc)
|
||||
|
||||
return len(stuck) + len(poison_ids) + orphan_count
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user