feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -23,6 +23,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from .discord_ingester import DiscordIngester
|
||||
from .gallery_dl import DownloadResult, ErrorType
|
||||
from .ingest_core import DEFAULT_REVISIT_DAYS
|
||||
from .patreon_ingester import PatreonIngester
|
||||
@@ -31,9 +32,13 @@ from .platforms import known_platform_keys
|
||||
from .subscribestar_ingester import SubscribeStarIngester
|
||||
|
||||
# Platforms whose download + verify go through the native ingester rather than
|
||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
|
||||
# they migrate too.
|
||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
|
||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry) until it
|
||||
# migrates too. Discord joined in milestone 428.
|
||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"})
|
||||
|
||||
# Native platforms whose feed id IS the source URL, so there is nothing to
|
||||
# resolve: SubscribeStar's creator page, Discord's server/channel link.
|
||||
_URL_IS_FEED = frozenset({"subscribestar", "discord"})
|
||||
|
||||
|
||||
def _unsupported_platform_message(platform: str) -> str | None:
|
||||
@@ -67,6 +72,7 @@ def _native_ingester_cls(platform: str):
|
||||
return {
|
||||
"patreon": PatreonIngester,
|
||||
"subscribestar": SubscribeStarIngester,
|
||||
"discord": DiscordIngester,
|
||||
}[platform]
|
||||
|
||||
|
||||
@@ -126,10 +132,10 @@ async def _resolve_native_campaign_id(
|
||||
platform: str, url: str, cookies_path: str | None, overrides: dict,
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
|
||||
feed id IS the creator URL (no lookup → resolved None). Patreon resolves the
|
||||
campaign id from the vanity URL (resolved non-None when a lookup actually ran,
|
||||
so phase 3 caches it)."""
|
||||
if platform == "subscribestar":
|
||||
and Discord's feed id IS the source URL (no lookup → resolved None). Patreon
|
||||
resolves the campaign id from the vanity URL (resolved non-None when a lookup
|
||||
actually ran, so phase 3 caches it)."""
|
||||
if platform in _URL_IS_FEED:
|
||||
return url, None
|
||||
return await resolve_campaign_id_for_source(url, cookies_path, overrides)
|
||||
|
||||
@@ -252,6 +258,10 @@ async def verify_source_credential(
|
||||
from .subscribestar_ingester import verify_subscribestar_credential
|
||||
|
||||
return await verify_subscribestar_credential(url, cookies_path, config_overrides)
|
||||
if platform == "discord":
|
||||
from .discord_ingester import verify_discord_credential
|
||||
|
||||
return await verify_discord_credential(url, auth_token)
|
||||
from .patreon_ingester import verify_patreon_credential
|
||||
|
||||
return await verify_patreon_credential(url, cookies_path, config_overrides)
|
||||
|
||||
Reference in New Issue
Block a user