Patreon: enforce the backfill time-box mid-post (stop soft-limit overruns) #78

Merged
bvandeusen merged 2 commits from dev into main 2026-06-06 23:00:29 -04:00

2 Commits

Author SHA1 Message Date
bvandeusen 14c4dd1ea0 test(patreon): adjust deterministic clock for the new per-media should_stop read
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m3s
The mid-post time-box check (619e771) reads time.monotonic() once more per post,
so test_backfill_budget_cut_returns_partial_with_progress's discrete tick
sequence shifted — the >budget tick (200) landed on post1's first-item check
instead of post2's gate, cutting post1 to 0 files. Add the extra tick (20, still
under budget) so post1's item downloads, matching production where the gate and
the first should_stop are microseconds apart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 22:26:27 -04:00
bvandeusen 619e7712c2 fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Failing after 3m3s
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).

download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.

Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 22:20:05 -04:00