feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -13,7 +13,6 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -156,6 +155,14 @@ class DownloadResult:
|
||||
duration_seconds: float = 0.0
|
||||
started_at: str | None = None
|
||||
completed_at: str | None = None
|
||||
# Plan #704 — structured fields the NATIVE ingester populates directly (None
|
||||
# on the gallery-dl path, which keeps the regex-over-stdout route). When set,
|
||||
# phase 3 reads these instead of scraping the text it would otherwise have to
|
||||
# reconstruct: `run_stats` mirrors _compute_run_stats' shape; `cursor` is the
|
||||
# backfill checkpoint the ingester knows exactly (no parse_last_cursor).
|
||||
run_stats: dict | None = None
|
||||
cursor: str | None = None
|
||||
posts_processed: int = 0
|
||||
|
||||
|
||||
def _summarize_validation_failures(failures: list[dict]) -> str:
|
||||
@@ -172,20 +179,9 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
|
||||
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
|
||||
|
||||
|
||||
# The native Patreon ingester emits its pagination cursor as `Cursor: <token>`
|
||||
# — one line per fetched page — into the DownloadResult stdout (mirroring the
|
||||
# convention gallery-dl used before the #697 cutover, so the backfill lifecycle
|
||||
# stayed unchanged). The LAST such line is the furthest-progressed page, i.e.
|
||||
# the resume point. We scan both streams (be defensive) and take the final
|
||||
# match; download_service checkpoints it as the next chunk's resume_cursor.
|
||||
# Survives a time-boxed chunk too: the ingester emits the cursor for a page when
|
||||
# it STARTS it, so an interrupted walk still yields its last cursor.
|
||||
_CURSOR_RE = re.compile(r"Cursor:\s*(\S+)")
|
||||
|
||||
|
||||
def parse_last_cursor(stdout: str, stderr: str) -> str | None:
|
||||
matches = _CURSOR_RE.findall(f"{stdout or ''}\n{stderr or ''}")
|
||||
return matches[-1] if matches else None
|
||||
# (parse_last_cursor was removed in plan #704: the native ingester now carries
|
||||
# its checkpoint cursor as a structured DownloadResult.cursor field, so there is
|
||||
# no log text to scrape — and gallery-dl platforms never had a cursor.)
|
||||
|
||||
|
||||
class GalleryDLService:
|
||||
|
||||
Reference in New Issue
Block a user