feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -8,7 +8,6 @@ from backend.app.services.gallery_dl import (
|
||||
ErrorType,
|
||||
GalleryDLService,
|
||||
SourceConfig,
|
||||
parse_last_cursor,
|
||||
)
|
||||
|
||||
|
||||
@@ -316,28 +315,6 @@ def test_categorize_tier_limited_wins_over_partial(gdl):
|
||||
return_code=1, stdout=stdout, stderr=stderr,
|
||||
)
|
||||
assert etype == ErrorType.TIER_LIMITED
|
||||
|
||||
|
||||
# --- Cursor parsing (plan #689 / native ingester #697) ---------------------
|
||||
# parse_last_cursor is retained post-cutover: the native Patreon ingester emits
|
||||
# `Cursor: <token>` per page into DownloadResult.stdout and download_service
|
||||
# checkpoints the last one. (The gallery-dl Patreon `files`/`cursor` config and
|
||||
# the Mux yt-dlp Referer block were removed at the #697 cutover, along with the
|
||||
# tests that pinned them.)
|
||||
|
||||
|
||||
def test_parse_last_cursor_returns_last_match_across_streams():
|
||||
# One `Cursor: <token>` line per page; the last is the furthest-progressed
|
||||
# resume point.
|
||||
stderr = (
|
||||
"[patreon][debug] Cursor: 03:AAAA:bbb\n"
|
||||
"[urllib3] GET ...\n"
|
||||
"[patreon][debug] Cursor: 03:CCCC:ddd\n"
|
||||
)
|
||||
assert parse_last_cursor("", stderr) == "03:CCCC:ddd"
|
||||
|
||||
|
||||
def test_parse_last_cursor_scans_stdout_too_and_none_when_absent():
|
||||
assert parse_last_cursor("Cursor: 99:ZZZ", "") == "99:ZZZ"
|
||||
assert parse_last_cursor("no marker here", "still nothing") is None
|
||||
assert parse_last_cursor("", "") is None
|
||||
# (The cursor-parsing tests were removed in plan #704: parse_last_cursor is gone
|
||||
# — the native ingester now carries its checkpoint cursor as a structured
|
||||
# DownloadResult.cursor field instead of a `Cursor:` log line to scrape.)
|
||||
|
||||
Reference in New Issue
Block a user