refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m0s

Consolidates duplication that owning the native ingester left against the still-
live gallery-dl path, and fixes a parity gap the duplication hid.

A1 — shared quarantine: extract file_validator.quarantine_file (move to
_quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar).
gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both
call it. PARITY FIX: the native path now writes the provenance sidecar it
previously skipped — threads the media url through for source_url.

A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats
key set; gallery_dl._compute_run_stats and ingest_core both build through it so
the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0).

A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for
the Path call sites + memory pointer) and patreon_client both use it. Closes the
double-impl of the URL-encoded-basename ext gotcha.

A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level
truncate_log/extract_errors_warnings; download_service calls them directly
instead of reaching through self.gdl for native-result log shaping. The
staticmethods stay as thin delegators for existing callers/tests.

Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine
writes a .quarantine.json (test_patreon_downloader).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 11:11:25 -04:00
parent 697a86d31c
commit b211900390
9 changed files with 222 additions and 137 deletions
+9 -11
View File
@@ -32,7 +32,7 @@ from collections.abc import Callable
from sqlalchemy import delete, func, select, text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from .gallery_dl import DownloadResult, ErrorType
from .gallery_dl import DownloadResult, ErrorType, make_run_stats
log = logging.getLogger(__name__)
@@ -153,16 +153,14 @@ class Ingester:
duration_seconds=time.monotonic() - start,
cursor=emitted_cursor,
posts_processed=posts_processed,
run_stats={
"exit_code": return_code,
"downloaded_count": downloaded,
"skipped_count": skipped_count,
"per_item_failures": errors,
"warning_count": 0,
"tier_gated_count": 0,
"quarantined_count": quarantined,
"dead_lettered_count": dead_lettered,
},
run_stats=make_run_stats(
exit_code=return_code,
downloaded_count=downloaded,
skipped_count=skipped_count,
per_item_failures=errors,
quarantined_count=quarantined,
dead_lettered_count=dead_lettered,
),
)
try: