fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit)
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked between POSTS, but download_post downloads ALL of one post's media synchronously — so a single media-heavy post could run the chunk far past 600s, all the way to the Celery soft time limit (1350s), where it was killed and finalized as error (Pocketacer, event #41330: ran the full 22.5 min). download_post now polls a should_stop() deadline BEFORE each media item and the engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the budget and the remaining media (never marked seen) re-fetch next chunk. Bounds chunk overrun to one media download instead of one whole post. Also genericized the soft-limit salvage message — it claimed the "gallery-dl subprocess" failed, which is wrong for a native Patreon walk; it now describes the time-budget overrun + per-page checkpoint resume in platform-neutral terms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -397,6 +397,25 @@ def test_skips_do_not_pace(tmp_path, monkeypatch):
|
||||
assert slept == []
|
||||
|
||||
|
||||
def test_download_post_honors_should_stop_mid_post(tmp_path):
|
||||
"""The time-box is polled BEFORE each media item, so a heavy post can't run
|
||||
the whole chunk past its budget (Pocketacer soft-limit, 2026-06-07)."""
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
session=_FakeSession(),
|
||||
)
|
||||
items = [_img("a.png"), _img("b.png"), _img("c.png")]
|
||||
polls = {"n": 0}
|
||||
|
||||
def _stop():
|
||||
polls["n"] += 1
|
||||
return polls["n"] > 1 # allow the first item, stop before the second
|
||||
|
||||
outcomes = dl.download_post(_post(), items, "artist-x", should_stop=_stop)
|
||||
assert len(outcomes) == 1 # only the first item ran; the rest re-fetch later
|
||||
assert outcomes[0].status == "downloaded"
|
||||
|
||||
|
||||
def test_media_429_retried_then_succeeds(tmp_path, monkeypatch):
|
||||
slept: list[float] = []
|
||||
monkeypatch.setattr(pd_mod.time, "sleep", lambda s: slept.append(s))
|
||||
|
||||
Reference in New Issue
Block a user