Discord was the last focus platform still on gallery-dl. This adds the native
path, mirrored from gallery-dl 1.32.13's discord extractor:
- discord_client: API v10 with the user token and gallery-dl's request
profile (dated Firefox UA, Referer). Walks a server, category, forum,
channel or thread in gallery-dl's order and pages each channel newest-first.
Files are attachments, then embeds, then forwards, numbered across the
message. The resume cursor is <channel>:<before>. Text-only messages are
not posts, since gallery-dl never made them.
- discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans
names on Linux (only `/` and control characters change), so existing files
are skipped_disk rather than fetched again. Sidecars carry identity only.
The message record keeps gallery-dl's keys, so parse_sidecar,
derive_post_url and the drop grouping read it unchanged.
- The ledger keys on the attachment id (or a hash of an embed's URL path),
not the file's position, which an edit can renumber. Migration 0111.
- DiscordIngester: token auth, body canary off (files-only drops are
normal). Registered as native, verified by token, and serialised
per-platform, since every source shares one user token.
- ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out
on a multi-channel source now ends the quiet channel, not the whole walk.
Clients without the seam behave as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.
- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
_NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
/ preview_source / verify_source_credential platform-aware. SubscribeStar's
campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
recover/recapture actions show for subscribestar; download_service's
cursor/mode/post_first + the preview endpoint already key on
uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
verify, gallery_dl_service skip-value, api_sources arm-no-preflight).
post_is_gated stays best-effort (can't cause junk downloads); not gating this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5