4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	0fad744bfb	fix: a restart no longer strands downloads — the download lane closes its predecessor's runs as interrupted and frees the platform locks at boot (#4433) ... A walk that outlives the 90s stop grace is SIGKILLed: its event never finalizes and its platform lock is held for the 27-min TTL. The 30-min sweep then errored every stranded event and bumped consecutive_failures, backing sources off (and blocking backfills) as if the platform failed. On worker_ready, the process consuming 'download' ends pre-boot pending/running events as skipped with error_type 'interrupted', leaves the source untouched so the next tick resumes it, and releases the serialized platforms' locks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:29:05 -04:00
bvandeusenandClaude Opus 5.5	84e5448941	feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428) ... Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:42:00 -04:00
bvandeusenandClaude Opus 4.8	d8d8ecd78f	feat(subscribestar): flip dispatch to the native ingester (#893, Step 5) ... SubscribeStar now downloads + verifies through the native core ingester instead of gallery-dl — the go-live switch for milestone #71. - download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester / preview_source / verify_source_credential platform-aware. SubscribeStar's campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity. preview now catches the shared NativeIngestError (covers both platforms). - platform_lock: subscribestar serialized (one paced walk at a time). - gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no fallback once native works). - frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the recover/recapture actions show for subscribestar; download_service's cursor/mode/post_first + the preview endpoint already key on uses_native_ingester, so backfill/recovery/recapture/preview light up for free. - tests: download_backends (subscribestar native), platform_lock (serialized), and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials verify, gallery_dl_service skip-value, api_sources arm-no-preflight). post_is_gated stays best-effort (can't cause junk downloads); not gating this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 12:03:48 -04:00
bvandeusenandClaude Opus 4.8	87c7318125	feat(downloads): serialize same-platform downloads (Patreon concurrency cap) ... Two concurrent Patreon walks could trip the server rate limit even with each source pacing its own requests. The platform-cooldown handled the aftermath of a 429; this adds the preventive half — a per-platform Redis lock so only one Patreon walk runs at a time. Different platforms still run concurrently up to the worker concurrency; only a second walk on the SAME serialized platform waits. download_source acquires fc:download_lock:<platform> (non-blocking) before the run. On contention it re-enqueues itself with a short countdown (the pending event stays — no new event, no log spam), bounded to ~15 min then runs uncapped as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd worker auto-releases; a backfill chunk only holds it ~10 min, well under the 30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped (prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon}; gallery-dl platforms are left uncapped (self-pacing subprocesses). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 21:22:32 -04:00