fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Failing after 3m3s

A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).

download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.

Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 22:20:05 -04:00
parent 416d8d71cd
commit 619e7712c2
5 changed files with 47 additions and 9 deletions
+10 -4
View File
@@ -9,9 +9,11 @@ construction by a thin adapter (e.g. `PatreonIngester`):
- `client` — `.iter_posts(feed_id, cursor)` yielding `(post, included,
page_cursor)` + `.extract_media(post, included) -> [media]`.
- `downloader`— `.download_post(post, media, artist_slug, is_seen) ->
[MediaOutcome]` (status in downloaded/skipped_seen/skipped_disk
/quarantined/error; `.path`/`.error`/`.post_id`).
- `downloader`— `.download_post(post, media, artist_slug, is_seen,
should_stop) -> [MediaOutcome]` (status in downloaded/
skipped_seen/skipped_disk/quarantined/error;
`.path`/`.error`/`.post_id`). `should_stop()` is polled
between media so the time-box is honoured mid-post.
- ledger — `seen_model` + `failed_model` SQLAlchemy models (+ their
on-conflict UNIQUE constraint names) and a `ledger_key(media)`.
- failure map — the adapter overrides `_failure_result` (platform exception
@@ -237,8 +239,12 @@ class Ingester:
def _is_skip(m, _skip=skip) -> bool:
return ledger_key(m) in _skip
# Honour the time-box DURING a media-dense post too, not only at
# the per-post boundary below — else one heavy post can blow the
# chunk budget out to the Celery soft limit (Pocketacer, 2026-06-07).
outcomes = self.downloader.download_post(
post, media, artist_slug, is_seen=_is_skip
post, media, artist_slug, is_seen=_is_skip,
should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
)
to_mark: list[tuple[str, str]] = []