refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,9 +11,15 @@ expected to write.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
JPEG_HEAD = b"\xff\xd8\xff"
|
||||
JPEG_TAIL = b"\xff\xd9"
|
||||
|
||||
@@ -108,3 +114,52 @@ def validate_file(path: Path) -> ValidationResult:
|
||||
return ValidationResult(ok=True, format="webp", size=size)
|
||||
|
||||
return ValidationResult(ok=True, format=None, size=size)
|
||||
|
||||
|
||||
def quarantine_file(
|
||||
images_root: Path,
|
||||
path: Path,
|
||||
artist_slug: str,
|
||||
platform: str,
|
||||
*,
|
||||
url: str | None,
|
||||
result: ValidationResult,
|
||||
) -> Path | None:
|
||||
"""Move a validation-failed file to `_quarantine/<slug>/<platform>` and write
|
||||
a `.quarantine.json` provenance sidecar next to it.
|
||||
|
||||
Returns the destination path, or None if the move itself failed (file left
|
||||
in place — the caller decides what to report). The caller has already run
|
||||
`validate_file` and seen `result.ok is False`. ONE implementation for both
|
||||
download backends — gallery-dl's batch post-process and the native ingester's
|
||||
per-media path — so the quarantine layout + provenance sidecar can't drift.
|
||||
"""
|
||||
quarantine_root = images_root / "_quarantine" / artist_slug / platform
|
||||
try:
|
||||
quarantine_root.mkdir(parents=True, exist_ok=True)
|
||||
dest = quarantine_root / path.name
|
||||
counter = 1
|
||||
while dest.exists():
|
||||
dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
|
||||
counter += 1
|
||||
shutil.move(str(path), str(dest))
|
||||
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
|
||||
sidecar.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"original_path": str(path),
|
||||
"source_url": url,
|
||||
"artist_slug": artist_slug,
|
||||
"platform": platform,
|
||||
"format": result.format,
|
||||
"reason": result.reason,
|
||||
"size": result.size,
|
||||
"quarantined_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
except OSError as exc:
|
||||
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
|
||||
return None
|
||||
return dest
|
||||
|
||||
Reference in New Issue
Block a user