Patreon download concurrency cap + immediate backfill kickoff on create #76

2026-06-06T21:27:16-04:00

bvandeusen commented

2026-06-06 21:27:16 -04:00

All CI-green on dev (run 632).

Per-platform concurrency cap (87c7318): a per-platform Redis lock serializes Patreon downloads to one walk at a time (other platforms still run concurrently). On contention the task re-enqueues itself (bounded ~15 min, then uncapped as a safety valve); the lock TTL auto-releases a SIGKILL'd worker; a broker hiccup degrades to uncapped. The preventive half that complements the existing 429 platform-cooldown.
Immediate backfill kickoff (a3c9499): a new enabled source fires its first walk right away (pending event + dispatch) instead of waiting up to ~60s for the next scheduler tick — skipped during a platform cooldown; disabled sources don't dispatch.

🤖 Generated with Claude Code

All CI-green on dev (run 632). - **Per-platform concurrency cap** (`87c7318`): a per-platform Redis lock serializes Patreon downloads to one walk at a time (other platforms still run concurrently). On contention the task re-enqueues itself (bounded ~15 min, then uncapped as a safety valve); the lock TTL auto-releases a SIGKILL'd worker; a broker hiccup degrades to uncapped. The preventive half that complements the existing 429 platform-cooldown. - **Immediate backfill kickoff** (`a3c9499`): a new enabled source fires its first walk right away (pending event + dispatch) instead of waiting up to ~60s for the next scheduler tick — skipped during a platform cooldown; disabled sources don't dispatch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-06 21:27:16 -04:00

feat(downloads): serialize same-platform downloads (Patreon concurrency cap) 87c7318125

Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.

download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(subs): kick off the first backfill walk immediately on source create

CI / lint (push) Successful in 3s

Details

CI / frontend-build (push) Successful in 24s

Details

CI / backend-lint-and-test (push) Successful in 26s

Details

CI / integration (push) Successful in 3m3s

Details

a3c9499e93

A new enabled source is armed for run-until-done backfill (#693) but would sit
idle until the next scheduler tick (~60s). create_source now enqueues the first
walk right away (pending DownloadEvent + download_source.delay), skipping only
when the platform is in a rate-limit cooldown (the scheduler picks it up when
that clears). Disabled sources still don't dispatch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit 3bb94674cf into main

2026-06-06 21:27:22 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#76