refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,8 @@ from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import requests
|
||||
|
||||
from ..utils.paths import safe_ext
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_POSTS_URL = "https://www.patreon.com/api/posts"
|
||||
@@ -99,11 +101,6 @@ _FILEHASH_RE = re.compile(r"([0-9a-fA-F]{32})")
|
||||
# by filehash. Tolerant of attribute ordering and single/double quotes.
|
||||
_CONTENT_IMG_RE = re.compile(r"<img\b[^>]*?\bsrc=[\"']([^\"']+)[\"']", re.IGNORECASE)
|
||||
|
||||
# A bounded, sane extension parse (see importer._safe_ext for the FC gotcha:
|
||||
# URL-encoded basenames make Path.suffix return base64-ish junk). We only ever
|
||||
# use this for a fallback filename, never a network call.
|
||||
_MAX_EXT_LEN = 16
|
||||
|
||||
|
||||
class PatreonAPIError(Exception):
|
||||
"""Base for native Patreon client failures.
|
||||
@@ -187,15 +184,6 @@ def _filehash(url: str) -> str | None:
|
||||
return match.group(1).lower() if match else None
|
||||
|
||||
|
||||
def _safe_ext(name: str) -> str:
|
||||
suffix = Path(name).suffix.lower()
|
||||
if not suffix or len(suffix) > _MAX_EXT_LEN:
|
||||
return ""
|
||||
if not all(c.isalnum() for c in suffix[1:]):
|
||||
return ""
|
||||
return suffix
|
||||
|
||||
|
||||
def _basename_from_url(url: str) -> str:
|
||||
"""Derive a sane filename from a URL when the media has no file_name.
|
||||
|
||||
@@ -206,7 +194,7 @@ def _basename_from_url(url: str) -> str:
|
||||
path = urlsplit(url).path
|
||||
base = os.path.basename(path)
|
||||
if base:
|
||||
ext = _safe_ext(base)
|
||||
ext = safe_ext(base)
|
||||
stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
|
||||
# Keep the stem bounded; URL-encoded stems can be enormous.
|
||||
stem = stem[:120] or "file"
|
||||
|
||||
Reference in New Issue
Block a user