feat(download): cursor-paged Patreon backfill for large catalogs
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 25s
CI / integration (push) Successful in 2m57s

Large Patreon creators (Anduo: weekly 50-120-image Reports back months =
thousands of files) couldn't backfill: each run re-walked newest→oldest
from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone
exceeded the 1170s subprocess budget, so the run died during enumeration
with 0 files written and NO forward progress — re-stranding every time
(event #40411).

Checkpoint gallery-dl's pagination cursor so each backfill window advances
the frontier:

- gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets
  extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True
  for a fresh run); parse_last_cursor() pulls the last emitted
  'Cursor: <token>' from stdout+stderr — survives a timed-out run since the
  TimeoutExpired path returns partial output.
- download_service.py: phase2 stays in BACKFILL mode while a cursor is
  pending (even after the run budget drains) and threads resume_cursor;
  _apply_backfill_lifecycle() checkpoints the advancing cursor each
  non-completing run, completes on a clean rc=0 finish (walk reached
  bottom), and a stuck-guard clears the cursor after 2 non-advancing runs
  so a wedged walk can't re-strand forever.

patreon-only (sole platform with a resumable cursor); other platforms keep
the simple counter semantics. Cursor state lives in config_overrides JSON
(patreon_campaign_id precedent) — no migration. Time-budget ladder
(1170/1350/1500) unchanged.

Plan #689.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-04 23:44:31 -04:00
parent 86efbf7f2c
commit 593f65c9cc
4 changed files with 297 additions and 40 deletions
+36
View File
@@ -8,6 +8,7 @@ from backend.app.services.gallery_dl import (
ErrorType,
GalleryDLService,
SourceConfig,
parse_last_cursor,
)
@@ -326,3 +327,38 @@ def test_default_config_forwards_patreon_referer_to_ytdl(gdl):
headers = cfg["downloader"]["ytdl"]["raw-options"]["http_headers"]
assert headers["Referer"] == "https://www.patreon.com/"
assert headers["Origin"] == "https://www.patreon.com"
# --- Cursor-paged backfill (plan #689) -------------------------------------
def test_parse_last_cursor_returns_last_match_across_streams():
# gallery-dl logs `Cursor: <token>` per page (debug → stderr); the last
# is the furthest-progressed resume point.
stderr = (
"[patreon][debug] Cursor: 03:AAAA:bbb\n"
"[urllib3] GET ...\n"
"[patreon][debug] Cursor: 03:CCCC:ddd\n"
)
assert parse_last_cursor("", stderr) == "03:CCCC:ddd"
def test_parse_last_cursor_scans_stdout_too_and_none_when_absent():
assert parse_last_cursor("Cursor: 99:ZZZ", "") == "99:ZZZ"
assert parse_last_cursor("no marker here", "still nothing") is None
assert parse_last_cursor("", "") is None
def test_build_config_patreon_resume_cursor_overrides_default(gdl):
# No resume cursor → PLATFORM_DEFAULTS leaves the log-only `cursor: True`.
fresh = gdl._build_config_for_source(
"patreon", SourceConfig(), "alice", skip_value=True,
)
assert fresh["extractor"]["patreon"]["cursor"] is True
# Resume cursor set → gallery-dl restarts the walk from that page.
resumed = gdl._build_config_for_source(
"patreon", SourceConfig(resume_cursor="03:CCCC:ddd"), "alice",
skip_value=True,
)
assert resumed["extractor"]["patreon"]["cursor"] == "03:CCCC:ddd"