refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -31,7 +31,12 @@ from ..models import (
|
||||
Source,
|
||||
)
|
||||
from ..utils import safe_probe
|
||||
from ..utils.paths import derive_subdir, derive_top_level_artist, hash_suffixed_name
|
||||
from ..utils.paths import (
|
||||
derive_subdir,
|
||||
derive_top_level_artist,
|
||||
hash_suffixed_name,
|
||||
safe_ext,
|
||||
)
|
||||
from ..utils.phash import compute_phash, find_similar
|
||||
from ..utils.sidecar import find_sidecar, parse_sidecar
|
||||
from ..utils.slug import slugify
|
||||
@@ -82,27 +87,9 @@ def is_video(path: Path) -> bool:
|
||||
|
||||
def _safe_ext(path: Path) -> str:
|
||||
"""Conservatively extract a file extension for PostAttachment.ext
|
||||
(varchar(32)).
|
||||
|
||||
gallery-dl produces some filenames with URL-encoded query-string
|
||||
artifacts embedded into the basename (e.g.
|
||||
`79507046_media_..._https___www.patreon.com_media-u_Z0FBQUFBQm5q...`).
|
||||
`Path.suffix` finds the LAST dot and returns everything after, which
|
||||
in those cases yields a 50+ char "extension" of mostly base64-ish
|
||||
junk. That blows the column. Operator-flagged 2026-05-25.
|
||||
|
||||
Real extensions are short and alphanumeric. We accept anything ≤ 16
|
||||
chars where every post-dot character is alphanumeric; anything else
|
||||
means the input wasn't a real extension and we return the empty
|
||||
string. ext is nullable-ish (empty string still satisfies NOT NULL)
|
||||
and consumers should treat "" as "no known extension".
|
||||
"""
|
||||
suffix = path.suffix.lower()
|
||||
if not suffix or len(suffix) > 16:
|
||||
return ""
|
||||
if not all(c.isalnum() for c in suffix[1:]):
|
||||
return ""
|
||||
return suffix
|
||||
(varchar(32)). Thin wrapper over the shared `utils.paths.safe_ext` (kept for
|
||||
the Path-typed call sites + the [[path_suffix_sanitize]] memory pointer)."""
|
||||
return safe_ext(path)
|
||||
|
||||
|
||||
def _mime_for(path: Path) -> str:
|
||||
|
||||
Reference in New Issue
Block a user