fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit)
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked between POSTS, but download_post downloads ALL of one post's media synchronously — so a single media-heavy post could run the chunk far past 600s, all the way to the Celery soft time limit (1350s), where it was killed and finalized as error (Pocketacer, event #41330: ran the full 22.5 min). download_post now polls a should_stop() deadline BEFORE each media item and the engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the budget and the remaining media (never marked seen) re-fetch next chunk. Bounds chunk overrun to one media download instead of one whole post. Also genericized the soft-limit salvage message — it claimed the "gallery-dl subprocess" failed, which is wrong for a native Patreon walk; it now describes the time-budget overrun + per-page checkpoint resume in platform-neutral terms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -94,10 +94,11 @@ def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
|
||||
ev.finished_at = now
|
||||
ev.error = (
|
||||
f"killed by Celery soft time limit ({DOWNLOAD_SOFT_TIME_LIMIT}s) "
|
||||
"before the gallery-dl subprocess returned — the run exceeded its "
|
||||
"budget and its stdout/stderr were lost with the worker thread. "
|
||||
"If this recurs, the source is too large for one run; the backfill "
|
||||
"budget was decremented so the next tick walks less."
|
||||
"before the download finished — the run exceeded its time budget. "
|
||||
"Progress is checkpointed per page, so the next tick resumes near "
|
||||
"the cut; the backfill budget was decremented so it walks less. If "
|
||||
"this recurs, a single post is heavy enough to overrun the chunk "
|
||||
"time-box on its own."
|
||||
)
|
||||
ev.metadata_ = {
|
||||
**(ev.metadata_ or {}),
|
||||
|
||||
Reference in New Issue
Block a user