bvandeusen
b211900390
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
62 lines
2.1 KiB
Python
62 lines
2.1 KiB
Python
"""Filesystem path helpers — destination derivation, hash-suffixed names."""
|
|
|
|
from pathlib import Path
|
|
|
|
_MAX_EXT_LEN = 16
|
|
|
|
|
|
def safe_ext(name: str | Path) -> str:
|
|
"""Conservatively extract a short, alphanumeric file extension.
|
|
|
|
gallery-dl and Patreon CDN URLs produce basenames with URL-encoded
|
|
query-string artifacts, so `Path.suffix` can return 50+ chars of base64-ish
|
|
junk that blows bounded VARCHAR columns (e.g. PostAttachment.ext varchar(32)).
|
|
Accept only a suffix ≤16 chars whose post-dot characters are all alphanumeric;
|
|
otherwise return "" (no known extension). Operator-flagged 2026-05-25 — ONE
|
|
impl for the importer and the native Patreon client.
|
|
"""
|
|
suffix = Path(name).suffix.lower()
|
|
if not suffix or len(suffix) > _MAX_EXT_LEN:
|
|
return ""
|
|
if not all(c.isalnum() for c in suffix[1:]):
|
|
return ""
|
|
return suffix
|
|
|
|
|
|
def derive_subdir(source_path: Path, import_root: Path) -> str:
|
|
"""Returns the relative subdirectory of source_path under import_root.
|
|
|
|
The top-level folder name is treated as the 'artist' bucket. Nested
|
|
paths preserve hierarchy.
|
|
|
|
import_root=/import
|
|
source_path=/import/Alice/sub/x.png -> "Alice/sub"
|
|
source_path=/import/Alice/x.png -> "Alice"
|
|
source_path=/import/x.png -> ""
|
|
"""
|
|
try:
|
|
rel = source_path.parent.relative_to(import_root)
|
|
except ValueError:
|
|
return ""
|
|
return str(rel) if str(rel) != "." else ""
|
|
|
|
|
|
def hash_suffixed_name(stem: str, sha256_hex: str, ext: str) -> str:
|
|
"""Builds 'stem__<first10ofhash><ext>'.
|
|
|
|
Examples:
|
|
hash_suffixed_name("photo", "abcdef1234567890...", ".png")
|
|
-> "photo__abcdef1234.png"
|
|
"""
|
|
return f"{stem}__{sha256_hex[:10]}{ext}"
|
|
|
|
|
|
def derive_top_level_artist(source_path: Path, import_root: Path) -> str | None:
|
|
"""Returns the top-level folder name under import_root, or None if the
|
|
file is directly in import_root.
|
|
"""
|
|
subdir = derive_subdir(source_path, import_root)
|
|
if not subdir:
|
|
return None
|
|
return subdir.split("/", 1)[0]
|