feat(download): cursor-paged Patreon backfill for large catalogs
Large Patreon creators (Anduo: weekly 50-120-image Reports back months = thousands of files) couldn't backfill: each run re-walked newest→oldest from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone exceeded the 1170s subprocess budget, so the run died during enumeration with 0 files written and NO forward progress — re-stranding every time (event #40411). Checkpoint gallery-dl's pagination cursor so each backfill window advances the frontier: - gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True for a fresh run); parse_last_cursor() pulls the last emitted 'Cursor: <token>' from stdout+stderr — survives a timed-out run since the TimeoutExpired path returns partial output. - download_service.py: phase2 stays in BACKFILL mode while a cursor is pending (even after the run budget drains) and threads resume_cursor; _apply_backfill_lifecycle() checkpoints the advancing cursor each non-completing run, completes on a clean rc=0 finish (walk reached bottom), and a stuck-guard clears the cursor after 2 non-advancing runs so a wedged walk can't re-strand forever. patreon-only (sole platform with a resumable cursor); other platforms keep the simple counter semantics. Cursor state lives in config_overrides JSON (patreon_campaign_id precedent) — no migration. Time-budget ladder (1170/1350/1500) unchanged. Plan #689. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -8,6 +8,7 @@ from backend.app.services.gallery_dl import (
|
||||
ErrorType,
|
||||
GalleryDLService,
|
||||
SourceConfig,
|
||||
parse_last_cursor,
|
||||
)
|
||||
|
||||
|
||||
@@ -326,3 +327,38 @@ def test_default_config_forwards_patreon_referer_to_ytdl(gdl):
|
||||
headers = cfg["downloader"]["ytdl"]["raw-options"]["http_headers"]
|
||||
assert headers["Referer"] == "https://www.patreon.com/"
|
||||
assert headers["Origin"] == "https://www.patreon.com"
|
||||
|
||||
|
||||
# --- Cursor-paged backfill (plan #689) -------------------------------------
|
||||
|
||||
|
||||
def test_parse_last_cursor_returns_last_match_across_streams():
|
||||
# gallery-dl logs `Cursor: <token>` per page (debug → stderr); the last
|
||||
# is the furthest-progressed resume point.
|
||||
stderr = (
|
||||
"[patreon][debug] Cursor: 03:AAAA:bbb\n"
|
||||
"[urllib3] GET ...\n"
|
||||
"[patreon][debug] Cursor: 03:CCCC:ddd\n"
|
||||
)
|
||||
assert parse_last_cursor("", stderr) == "03:CCCC:ddd"
|
||||
|
||||
|
||||
def test_parse_last_cursor_scans_stdout_too_and_none_when_absent():
|
||||
assert parse_last_cursor("Cursor: 99:ZZZ", "") == "99:ZZZ"
|
||||
assert parse_last_cursor("no marker here", "still nothing") is None
|
||||
assert parse_last_cursor("", "") is None
|
||||
|
||||
|
||||
def test_build_config_patreon_resume_cursor_overrides_default(gdl):
|
||||
# No resume cursor → PLATFORM_DEFAULTS leaves the log-only `cursor: True`.
|
||||
fresh = gdl._build_config_for_source(
|
||||
"patreon", SourceConfig(), "alice", skip_value=True,
|
||||
)
|
||||
assert fresh["extractor"]["patreon"]["cursor"] is True
|
||||
|
||||
# Resume cursor set → gallery-dl restarts the walk from that page.
|
||||
resumed = gdl._build_config_for_source(
|
||||
"patreon", SourceConfig(resume_cursor="03:CCCC:ddd"), "alice",
|
||||
skip_value=True,
|
||||
)
|
||||
assert resumed["extractor"]["patreon"]["cursor"] == "03:CCCC:ddd"
|
||||
|
||||
Reference in New Issue
Block a user