Large Patreon creators (Anduo: weekly 50-120-image Reports back months =
thousands of files) couldn't backfill: each run re-walked newest→oldest
from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone
exceeded the 1170s subprocess budget, so the run died during enumeration
with 0 files written and NO forward progress — re-stranding every time
(event #40411).
Checkpoint gallery-dl's pagination cursor so each backfill window advances
the frontier:
- gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets
extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True
for a fresh run); parse_last_cursor() pulls the last emitted
'Cursor: <token>' from stdout+stderr — survives a timed-out run since the
TimeoutExpired path returns partial output.
- download_service.py: phase2 stays in BACKFILL mode while a cursor is
pending (even after the run budget drains) and threads resume_cursor;
_apply_backfill_lifecycle() checkpoints the advancing cursor each
non-completing run, completes on a clean rc=0 finish (walk reached
bottom), and a stuck-guard clears the cursor after 2 non-advancing runs
so a wedged walk can't re-strand forever.
patreon-only (sole platform with a resumable cursor); other platforms keep
the simple counter semantics. Cursor state lives in config_overrides JSON
(patreon_campaign_id precedent) — no migration. Time-budget ladder
(1170/1350/1500) unchanged.
Plan #689.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen
593f65c9cc