feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
"""Native Discord ingester — the Discord ADAPTER over `ingest_core.Ingester`.
|
||||
|
||||
Thin counterpart to subscribestar_ingester (milestone 428). The walk's modes,
|
||||
both ledgers, cursor checkpointing and the post-first capture live in the core;
|
||||
this wires in the Discord client, downloader, ledger models and key.
|
||||
|
||||
Two things differ from the cookie platforms:
|
||||
|
||||
- Discord authenticates with a user TOKEN, so `auth_token` is the credential
|
||||
here rather than an argument accepted and ignored.
|
||||
- The body canary is off. It fails a walk whose first 30+ captured posts all
|
||||
came back without text, on the theory that a creator nearly always writes
|
||||
something; a Discord drop is routinely files and nothing else, so on
|
||||
Discord that is an ordinary backfill, not a broken parser.
|
||||
|
||||
`campaign_id` is the source URL (a server, channel, thread or category link).
|
||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from ..models import DiscordFailedMedia, DiscordSeenMedia
|
||||
from .discord_client import DiscordAPIError, DiscordClient, MediaItem
|
||||
from .discord_downloader import DiscordDownloader
|
||||
from .ingest_core import Ingester
|
||||
|
||||
_LEDGER_KEY_MAX = 128
|
||||
|
||||
|
||||
def _ledger_key(media: MediaItem) -> str:
|
||||
"""`<message_id>:<media_id>` — stable across edits (see MediaItem)."""
|
||||
return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
|
||||
|
||||
|
||||
class DiscordIngester(Ingester):
|
||||
"""Walk a Discord source's channels, download unseen files, return a
|
||||
`DownloadResult`. `client` / `downloader` are injectable for tests."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
images_root: Path,
|
||||
cookies_path: str | None,
|
||||
session_factory: Callable[[], object],
|
||||
*,
|
||||
validate: bool = True,
|
||||
rate_limit: float = 0.0,
|
||||
request_sleep: float = 0.0,
|
||||
auth_token: str | None = None,
|
||||
client: DiscordClient | None = None,
|
||||
downloader: DiscordDownloader | None = None,
|
||||
):
|
||||
del cookies_path # Discord authenticates by token (uniform signature)
|
||||
self.images_root = Path(images_root)
|
||||
super().__init__(
|
||||
client=client if client is not None else DiscordClient(
|
||||
auth_token, request_sleep=request_sleep,
|
||||
),
|
||||
downloader=downloader if downloader is not None else DiscordDownloader(
|
||||
self.images_root, validate=validate, rate_limit=rate_limit,
|
||||
),
|
||||
session_factory=session_factory,
|
||||
seen_model=DiscordSeenMedia,
|
||||
failed_model=DiscordFailedMedia,
|
||||
seen_constraint="uq_discord_seen_media_source_id",
|
||||
failed_constraint="uq_discord_failed_media_source_id",
|
||||
ledger_key=_ledger_key,
|
||||
platform="discord",
|
||||
error_base=DiscordAPIError,
|
||||
drift_label="Discord API",
|
||||
body_canary=False,
|
||||
)
|
||||
|
||||
|
||||
async def verify_discord_credential(url: str, auth_token: str | None) -> tuple[bool | None, str]:
|
||||
"""The uniform `(ok, message)` probe: is the token valid, and can its
|
||||
account see the channel or server the source names?"""
|
||||
if not auth_token:
|
||||
return False, "No Discord token is saved — add one under Credentials."
|
||||
client = DiscordClient(auth_token)
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(None, client.verify_auth, url)
|
||||
Reference in New Issue
Block a user