feat(patreon): ingester rate-limit resilience — #703 step 1
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Failing after 3m0s

A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:

- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
  `rate_limit`, read off self.gdl) as a pre-download sleep on real media
  downloads only (skips don't pace); pace /api/posts page fetches with the
  per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
  the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
  429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
  30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
  RATE_LIMITED. Light 2-retry on a media-GET 429 too.

Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).

Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 23:13:46 -04:00
parent 218bfebb92
commit 3b2f7a41c3
6 changed files with 254 additions and 18 deletions
+15
View File
@@ -207,11 +207,26 @@ class DownloadService:
None,
)
# Honor the operator's existing rate-limit knobs on the native path
# (plan #703): the global download_rate_limit_seconds (gallery-dl's
# `rate_limit`, here on self.gdl) paces media downloads. Page fetches use
# the per-source sleep_request override, else `max(0.5, rate_limit/4)` —
# the SAME API-pacing default gallery-dl applied as its `sleep-request`
# (gallery_dl._get_default_config), so the rate-limited /api/posts
# endpoint stays politely paced without over-throttling.
rate_limit = self.gdl._rate_limit
request_sleep = (
source_config.sleep_request
if source_config.sleep_request is not None
else max(0.5, rate_limit / 4)
)
ingester = PatreonIngester(
images_root=self.gdl.images_root,
cookies_path=ctx["cookies_path"],
session_factory=self.sync_session_factory,
validate=self.gdl._validate_files,
rate_limit=rate_limit,
request_sleep=request_sleep,
)
loop = asyncio.get_running_loop()
dl_result = await loop.run_in_executor(