Patreon: enforce the backfill time-box mid-post (stop soft-limit overruns) #78

Merged

bvandeusen merged 2 commits from dev into main 2026-06-06 23:00:29 -04:00

Conversation 0 Commits 2 Files Changed 5

+54 -13

bvandeusen commented 2026-06-06 23:00:23 -04:00

Owner

Copy Link

Copy Source

All CI-green on dev (run 642).

A native Patreon backfill chunk's time-box (600s) was only checked between posts, but download_post downloads all of one post's media synchronously — so a media-heavy post (Pocketacer, event #41330) ran the chunk to the Celery soft limit (1350s) and finalized as error. Now the deadline is polled before each media item (should_stop()), so a heavy post stops at the budget and the remaining media re-fetch next chunk — overrun bounded to one media download instead of one whole post. Also genericized the soft-limit salvage message (it wrongly blamed the "gallery-dl subprocess" for a native walk).

619e771 fix + 14c4dd1 deterministic-clock test adjustment. Known residual (a single large streamed file can still overrun by its own download time) is documented in Scribe #723.

🤖 Generated with Claude Code

All CI-green on dev (run 642). A native Patreon backfill chunk's time-box (600s) was only checked between posts, but `download_post` downloads all of one post's media synchronously — so a media-heavy post (Pocketacer, event #41330) ran the chunk to the Celery soft limit (1350s) and finalized as error. Now the deadline is polled before each media item (`should_stop()`), so a heavy post stops at the budget and the remaining media re-fetch next chunk — overrun bounded to one media download instead of one whole post. Also genericized the soft-limit salvage message (it wrongly blamed the "gallery-dl subprocess" for a native walk). `619e771` fix + `14c4dd1` deterministic-clock test adjustment. Known residual (a single large *streamed* file can still overrun by its own download time) is documented in Scribe #723. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-06 23:00:23 -04:00

fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit) ... 619e7712c2

A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).

download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.

Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(patreon): adjust deterministic clock for the new per-media should_stop read ... 14c4dd1ea0

The mid-post time-box check (619e771) reads time.monotonic() once more per post,
so test_backfill_budget_cut_returns_partial_with_progress's discrete tick
sequence shifted — the >budget tick (200) landed on post1's first-item check
instead of post2's gate, cutting post1 to 0 files. Add the extra tick (20, still
under budget) so post1's item downloads, matching production where the gate and
the first should_stop are microseconds apart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit 2eb9fd5dd0 into main 2026-06-06 23:00:29 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#78