refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -33,6 +33,8 @@ from .gallery_dl import (
|
||||
ErrorType,
|
||||
GalleryDLService,
|
||||
SourceConfig,
|
||||
extract_errors_warnings,
|
||||
truncate_log,
|
||||
)
|
||||
from .importer import Importer
|
||||
from .patreon_ingester import PatreonIngester
|
||||
@@ -450,7 +452,7 @@ class DownloadService:
|
||||
dl_result.return_code, dl_result.stdout, dl_result.stderr
|
||||
)
|
||||
run_stats["quarantined_count"] = dl_result.files_quarantined
|
||||
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
|
||||
stderr_summary = extract_errors_warnings(dl_result.stderr)
|
||||
|
||||
# Plan #544: PARTIAL means the run downloaded ≥1 file but the
|
||||
# subprocess didn't finish in budget (typically wall-clock timeout
|
||||
@@ -470,8 +472,8 @@ class DownloadService:
|
||||
ev.metadata_ = {
|
||||
"run_stats": run_stats,
|
||||
"error_type": dl_result.error_type.value if dl_result.error_type else None,
|
||||
"stdout": self.gdl._truncate_log(dl_result.stdout) or None,
|
||||
"stderr": self.gdl._truncate_log(dl_result.stderr) or None,
|
||||
"stdout": truncate_log(dl_result.stdout) or None,
|
||||
"stderr": truncate_log(dl_result.stderr) or None,
|
||||
"stderr_errors_warnings": stderr_summary or None,
|
||||
"duration_seconds": dl_result.duration_seconds,
|
||||
"quarantined_paths": dl_result.quarantined_paths or None,
|
||||
|
||||
Reference in New Issue
Block a user