feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -33,7 +33,6 @@ from .gallery_dl import (
|
||||
ErrorType,
|
||||
GalleryDLService,
|
||||
SourceConfig,
|
||||
parse_last_cursor,
|
||||
)
|
||||
from .importer import Importer
|
||||
from .patreon_ingester import PatreonIngester
|
||||
@@ -155,7 +154,7 @@ class DownloadService:
|
||||
# timed out with NO progress stays TIMEOUT and feeds phase 3's
|
||||
# stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires.
|
||||
if in_backfill and dl_result.error_type == ErrorType.TIMEOUT:
|
||||
new_cursor = parse_last_cursor(dl_result.stdout, dl_result.stderr)
|
||||
new_cursor = dl_result.cursor # plan #704: structured, not scraped
|
||||
advanced = bool(
|
||||
(new_cursor and new_cursor != overrides.get("_backfill_cursor"))
|
||||
or dl_result.files_downloaded > 0
|
||||
@@ -439,9 +438,14 @@ class DownloadService:
|
||||
select(DownloadEvent).where(DownloadEvent.id == event_id)
|
||||
)).scalar_one()
|
||||
|
||||
run_stats = self.gdl._compute_run_stats(
|
||||
dl_result.return_code, dl_result.stdout, dl_result.stderr
|
||||
)
|
||||
# plan #704: the native ingester returns structured run_stats; only the
|
||||
# gallery-dl path needs the regex-over-stdout reconstruction.
|
||||
if dl_result.run_stats is not None:
|
||||
run_stats = dict(dl_result.run_stats)
|
||||
else:
|
||||
run_stats = self.gdl._compute_run_stats(
|
||||
dl_result.return_code, dl_result.stdout, dl_result.stderr
|
||||
)
|
||||
run_stats["quarantined_count"] = dl_result.files_quarantined
|
||||
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
|
||||
|
||||
@@ -528,12 +532,12 @@ class DownloadService:
|
||||
src.backfill_runs_remaining = 0
|
||||
return
|
||||
|
||||
# Did not finish. Patreon checkpoints + resumes via cursor; other
|
||||
# platforms have no resumable cursor (every chunk re-walks from the
|
||||
# top), so they advance only by the download archive growing.
|
||||
# Did not finish. The native ingester checkpoints + resumes via cursor
|
||||
# (carried structurally on the result, plan #704); gallery-dl platforms
|
||||
# have no resumable cursor (every chunk re-walks from the top), so they
|
||||
# advance only by the download archive growing.
|
||||
new_cursor = (
|
||||
parse_last_cursor(dl_result.stdout, dl_result.stderr)
|
||||
if uses_native_ingester(ctx["platform"]) else None
|
||||
dl_result.cursor if uses_native_ingester(ctx["platform"]) else None
|
||||
)
|
||||
advanced = bool(
|
||||
(new_cursor and new_cursor != old_cursor)
|
||||
|
||||
Reference in New Issue
Block a user