bvandeusen
87c7318125
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
60 lines
2.1 KiB
Python
60 lines
2.1 KiB
Python
"""Per-platform download concurrency cap.
|
|
|
|
Some platforms (Patreon) are API-rate-sensitive enough that two *simultaneous*
|
|
walks can trip the server's rate limit even with each source pacing its own
|
|
requests. The platform-cooldown handles the AFTERMATH of a 429; this is the
|
|
preventive half — it serializes downloads PER PLATFORM to one at a time.
|
|
|
|
Different platforms still run concurrently up to the worker's concurrency; only
|
|
a second walk on the SAME serialized platform waits. The lock lives in Redis
|
|
(the Celery broker) with a TTL, so a SIGKILL'd worker can't wedge a platform —
|
|
the lock auto-expires shortly after the download hard time limit.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
import redis
|
|
|
|
from ..config import get_config
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
# Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
|
|
# here: each runs as a self-pacing subprocess and they're lower-volume. Add a
|
|
# platform here to cap it to a single concurrent walk.
|
|
SERIALIZED_PLATFORMS = frozenset({"patreon"})
|
|
|
|
_LOCK_PREFIX = "fc:download_lock:"
|
|
|
|
_client: redis.Redis | None = None
|
|
|
|
|
|
def _redis() -> redis.Redis:
|
|
# One client per worker process (Celery prefork forks before tasks run, so
|
|
# each process lazily builds its own). redis-py pools connections.
|
|
global _client
|
|
if _client is None:
|
|
_client = redis.from_url(get_config().celery_broker_url)
|
|
return _client
|
|
|
|
|
|
def platform_lock(platform: str, *, ttl_seconds: int):
|
|
"""A non-blocking Redis lock for `platform`, or None when the platform is
|
|
not serialized. Caller does `.acquire(blocking=False)` / `.release()`.
|
|
|
|
Returns None (rather than raising) on any Redis error so a broker hiccup
|
|
degrades to the prior behaviour (uncapped) instead of stalling downloads.
|
|
"""
|
|
if platform not in SERIALIZED_PLATFORMS:
|
|
return None
|
|
try:
|
|
return _redis().lock(
|
|
f"{_LOCK_PREFIX}{platform}",
|
|
timeout=ttl_seconds,
|
|
blocking=False,
|
|
)
|
|
except redis.RedisError as exc: # pragma: no cover - broker outage
|
|
log.warning("platform_lock unavailable for %s: %s", platform, exc)
|
|
return None
|