refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -305,6 +305,28 @@ def test_validation_quarantines_corrupt(tmp_path):
|
||||
assert any(quarantine.iterdir())
|
||||
|
||||
|
||||
def test_quarantine_writes_provenance_sidecar(tmp_path):
|
||||
"""A1 parity fix: the native path now writes the same `.quarantine.json`
|
||||
provenance sidecar gallery-dl writes (it skipped it before the shared
|
||||
file_validator.quarantine_file helper)."""
|
||||
import json
|
||||
|
||||
bad = b"not a real png"
|
||||
url = "https://cdn.patreon.com/corrupt.png"
|
||||
session = _FakeSession({url: bad})
|
||||
dl = _downloader(tmp_path, session=session, validate=True)
|
||||
outcomes = dl.download_post(_post(), [_img("corrupt.png", url=url)], "artist-x")
|
||||
|
||||
dest = outcomes[0].path
|
||||
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
|
||||
assert sidecar.is_file()
|
||||
meta = json.loads(sidecar.read_text())
|
||||
assert meta["source_url"] == url
|
||||
assert meta["platform"] == "patreon"
|
||||
assert meta["artist_slug"] == "artist-x"
|
||||
assert meta["reason"]
|
||||
|
||||
|
||||
def test_validation_off_keeps_bytes(tmp_path):
|
||||
bad = b"not a real png"
|
||||
session = _FakeSession({"https://cdn.patreon.com/x.png": bad})
|
||||
|
||||
Reference in New Issue
Block a user