feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -165,6 +165,22 @@ def test_skip_seen(tmp_path):
|
||||
assert not (tmp_path / "artist-x").exists()
|
||||
|
||||
|
||||
def test_invalid_file_is_quarantined(tmp_path):
|
||||
"""plan #704 (#4): a downloaded file that fails validation is moved to
|
||||
_quarantine and reported as a 'quarantined' outcome carrying that path —
|
||||
distinct from a download 'error'."""
|
||||
bad_url = "https://cdn.patreon.com/bad.png"
|
||||
session = _FakeSession({bad_url: b"this is not a valid PNG"})
|
||||
dl = _downloader(tmp_path, session=session, validate=True)
|
||||
outcomes = dl.download_post(
|
||||
_post(), [_img("bad.png", url=bad_url)], "artist-x"
|
||||
)
|
||||
assert outcomes[0].status == "quarantined"
|
||||
assert outcomes[0].path is not None
|
||||
assert "_quarantine" in str(outcomes[0].path)
|
||||
assert outcomes[0].path.is_file()
|
||||
|
||||
|
||||
def test_skip_disk(tmp_path):
|
||||
dl = _downloader(tmp_path)
|
||||
post_dir = tmp_path / "artist-x" / "patreon" / "2026-05-01_1001_My Post Title"
|
||||
|
||||
Reference in New Issue
Block a user