feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Failing after 25s
CI / frontend-build (push) Successful in 37s
CI / integration (push) Failing after 3m0s

The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and
phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and
quarantine stats blank. We own the ingester, so it now RETURNS structured
data and phase 3 reads it directly.

- DownloadResult gains run_stats/cursor/posts_processed (None/0 on the
  gallery-dl path, which keeps the text route).
- Ingester builds real run_stats from per-media outcome counts, sets the
  checkpoint cursor structurally (no fake `Cursor:` stdout), and counts
  posts processed. download_service phase 3 uses dl_result.run_stats when
  present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint
  dl_result.cursor instead of parse_last_cursor(stdout).
- #4 quarantine: PatreonDownloader reports a distinct "quarantined"
  MediaOutcome (with the _quarantine dest); the ingester surfaces a real
  files_quarantined + quarantined_paths + run_stats.quarantined_count
  (was hardcoded 0). Quarantined media isn't written or marked seen.
- Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`)
  removed from gallery_dl — the structured cursor replaced the scrape.

Tests: ingester result carries real run_stats/cursor/posts_processed +
quarantine counts; downloader quarantines an invalid file as "quarantined";
backfill cursor tests pass cursor= structurally; dropped the
parse_last_cursor tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 23:39:28 -04:00
parent 5b615b7ded
commit e53f8959af
8 changed files with 162 additions and 86 deletions
+3 -26
View File
@@ -8,7 +8,6 @@ from backend.app.services.gallery_dl import (
ErrorType,
GalleryDLService,
SourceConfig,
parse_last_cursor,
)
@@ -316,28 +315,6 @@ def test_categorize_tier_limited_wins_over_partial(gdl):
return_code=1, stdout=stdout, stderr=stderr,
)
assert etype == ErrorType.TIER_LIMITED
# --- Cursor parsing (plan #689 / native ingester #697) ---------------------
# parse_last_cursor is retained post-cutover: the native Patreon ingester emits
# `Cursor: <token>` per page into DownloadResult.stdout and download_service
# checkpoints the last one. (The gallery-dl Patreon `files`/`cursor` config and
# the Mux yt-dlp Referer block were removed at the #697 cutover, along with the
# tests that pinned them.)
def test_parse_last_cursor_returns_last_match_across_streams():
# One `Cursor: <token>` line per page; the last is the furthest-progressed
# resume point.
stderr = (
"[patreon][debug] Cursor: 03:AAAA:bbb\n"
"[urllib3] GET ...\n"
"[patreon][debug] Cursor: 03:CCCC:ddd\n"
)
assert parse_last_cursor("", stderr) == "03:CCCC:ddd"
def test_parse_last_cursor_scans_stdout_too_and_none_when_absent():
assert parse_last_cursor("Cursor: 99:ZZZ", "") == "99:ZZZ"
assert parse_last_cursor("no marker here", "still nothing") is None
assert parse_last_cursor("", "") is None
# (The cursor-parsing tests were removed in plan #704: parse_last_cursor is gone
# — the native ingester now carries its checkpoint cursor as a structured
# DownloadResult.cursor field instead of a `Cursor:` log line to scrape.)