feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Failing after 25s
CI / frontend-build (push) Successful in 37s
CI / integration (push) Failing after 3m0s

The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and
phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and
quarantine stats blank. We own the ingester, so it now RETURNS structured
data and phase 3 reads it directly.

- DownloadResult gains run_stats/cursor/posts_processed (None/0 on the
  gallery-dl path, which keeps the text route).
- Ingester builds real run_stats from per-media outcome counts, sets the
  checkpoint cursor structurally (no fake `Cursor:` stdout), and counts
  posts processed. download_service phase 3 uses dl_result.run_stats when
  present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint
  dl_result.cursor instead of parse_last_cursor(stdout).
- #4 quarantine: PatreonDownloader reports a distinct "quarantined"
  MediaOutcome (with the _quarantine dest); the ingester surfaces a real
  files_quarantined + quarantined_paths + run_stats.quarantined_count
  (was hardcoded 0). Quarantined media isn't written or marked seen.
- Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`)
  removed from gallery_dl — the structured cursor replaced the scrape.

Tests: ingester result carries real run_stats/cursor/posts_processed +
quarantine counts; downloader quarantines an invalid file as "quarantined";
backfill cursor tests pass cursor= structurally; dropped the
parse_last_cursor tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 23:39:28 -04:00
parent 5b615b7ded
commit e53f8959af
8 changed files with 162 additions and 86 deletions
+14 -10
View File
@@ -33,7 +33,6 @@ from .gallery_dl import (
ErrorType,
GalleryDLService,
SourceConfig,
parse_last_cursor,
)
from .importer import Importer
from .patreon_ingester import PatreonIngester
@@ -155,7 +154,7 @@ class DownloadService:
# timed out with NO progress stays TIMEOUT and feeds phase 3's
# stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires.
if in_backfill and dl_result.error_type == ErrorType.TIMEOUT:
new_cursor = parse_last_cursor(dl_result.stdout, dl_result.stderr)
new_cursor = dl_result.cursor # plan #704: structured, not scraped
advanced = bool(
(new_cursor and new_cursor != overrides.get("_backfill_cursor"))
or dl_result.files_downloaded > 0
@@ -439,9 +438,14 @@ class DownloadService:
select(DownloadEvent).where(DownloadEvent.id == event_id)
)).scalar_one()
run_stats = self.gdl._compute_run_stats(
dl_result.return_code, dl_result.stdout, dl_result.stderr
)
# plan #704: the native ingester returns structured run_stats; only the
# gallery-dl path needs the regex-over-stdout reconstruction.
if dl_result.run_stats is not None:
run_stats = dict(dl_result.run_stats)
else:
run_stats = self.gdl._compute_run_stats(
dl_result.return_code, dl_result.stdout, dl_result.stderr
)
run_stats["quarantined_count"] = dl_result.files_quarantined
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
@@ -528,12 +532,12 @@ class DownloadService:
src.backfill_runs_remaining = 0
return
# Did not finish. Patreon checkpoints + resumes via cursor; other
# platforms have no resumable cursor (every chunk re-walks from the
# top), so they advance only by the download archive growing.
# Did not finish. The native ingester checkpoints + resumes via cursor
# (carried structurally on the result, plan #704); gallery-dl platforms
# have no resumable cursor (every chunk re-walks from the top), so they
# advance only by the download archive growing.
new_cursor = (
parse_last_cursor(dl_result.stdout, dl_result.stderr)
if uses_native_ingester(ctx["platform"]) else None
dl_result.cursor if uses_native_ingester(ctx["platform"]) else None
)
advanced = bool(
(new_cursor and new_cursor != old_cursor)