feat(patreon): ingester rate-limit resilience — #703 step 1
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:
- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
`rate_limit`, read off self.gdl) as a pre-download sleep on real media
downloads only (skips don't pace); pace /api/posts page fetches with the
per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
RATE_LIMITED. Light 2-retry on a media-GET 429 too.
Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).
Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -104,18 +104,28 @@ class PatreonIngester:
|
||||
session_factory: Callable[[], object],
|
||||
*,
|
||||
validate: bool = True,
|
||||
rate_limit: float = 0.0,
|
||||
request_sleep: float = 0.0,
|
||||
client: PatreonClient | None = None,
|
||||
downloader: PatreonDownloader | None = None,
|
||||
):
|
||||
self.images_root = Path(images_root)
|
||||
self.cookies_path = str(cookies_path) if cookies_path else None
|
||||
self.session_factory = session_factory
|
||||
self.client = client if client is not None else PatreonClient(cookies_path)
|
||||
# Pacing (plan #703): request_sleep paces the API page fetches,
|
||||
# rate_limit paces the media downloads. Injected client/downloader (in
|
||||
# tests) already carry their own pacing, so these only apply to the
|
||||
# default-constructed ones.
|
||||
self.client = (
|
||||
client
|
||||
if client is not None
|
||||
else PatreonClient(cookies_path, request_sleep=request_sleep)
|
||||
)
|
||||
self.downloader = (
|
||||
downloader
|
||||
if downloader is not None
|
||||
else PatreonDownloader(
|
||||
self.images_root, cookies_path, validate=validate
|
||||
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user