063778e6b3801fc4b9136d09cea636149566db34
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5.5
|
84e5448941 |
feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / integration (push) Successful in 2m27s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR |
||
|
bvandeusenandClaude Opus 5
|
ddf896078c |
refactor(platforms): retire deviantart end-to-end (#3069)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 23s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m43s
Executes the 2026-07-05 product decision (FC downloaders = art-dedicated
services only), which removed Twitter/X and Bluesky but left deviantart
fully wired for seven weeks — the half-retired state rule 22 exists to
prevent.
Removed: the PlatformInfo module and its registry entry, the gallery-dl
extractor block, extension_service's artist-page pattern, the extension's
PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission
and content-script match, the frontend icon/colour/label, and the operator-
facing "supported platforms" list that still advertised it.
Two judgment calls, both recorded in migration 0088:
* existing `source` rows are DISABLED, not deleted. The row is the only
record of the artist's DeviantArt URL. Disabling is also required for
correctness rather than tidiness: with the platform unregistered the
download path falls through to gallery-dl, which carries its OWN
deviantart extractor, so an enabled row would have kept downloading
from a dropped platform.
* the `credential` row IS deleted — a live session cookie for a site FC
will never call again.
Adds the invariant whose absence is why manifest.json drifted in the first
place: nothing tied its domain lists back to the platform table. The
extension suite now asserts both directions, plus that no host permission
belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at
an operator-chosen URL the extension cannot enumerate).
Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged
extension change without a bump. No release is cut — build.yml's
sign-extension job only runs on main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
d8d8ecd78f |
feat(subscribestar): flip dispatch to the native ingester (#893, Step 5)
SubscribeStar now downloads + verifies through the native core ingester instead of gallery-dl — the go-live switch for milestone #71. - download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester / preview_source / verify_source_credential platform-aware. SubscribeStar's campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity. preview now catches the shared NativeIngestError (covers both platforms). - platform_lock: subscribestar serialized (one paced walk at a time). - gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no fallback once native works). - frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the recover/recapture actions show for subscribestar; download_service's cursor/mode/post_first + the preview endpoint already key on uses_native_ingester, so backfill/recovery/recapture/preview light up for free. - tests: download_backends (subscribestar native), platform_lock (serialized), and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials verify, gallery_dl_service skip-value, api_sources arm-no-preflight). post_is_gated stays best-effort (can't cause junk downloads); not gating this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
87c7318125 |
feat(downloads): serialize same-platform downloads (Patreon concurrency cap)
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|