feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -8,6 +8,8 @@ from .backup_run import BackupRun
|
||||
from .base import Base
|
||||
from .character_prototype import CcipPrototypeState, CharacterPrototype
|
||||
from .credential import Credential
|
||||
from .discord_failed_media import DiscordFailedMedia
|
||||
from .discord_seen_media import DiscordSeenMedia
|
||||
from .download_event import DownloadEvent
|
||||
from .external_link import ExternalLink
|
||||
from .gpu_job import GpuJob
|
||||
@@ -56,6 +58,8 @@ __all__ = [
|
||||
"BackupRun",
|
||||
"Source",
|
||||
"Credential",
|
||||
"DiscordFailedMedia",
|
||||
"DiscordSeenMedia",
|
||||
"PatreonFailedMedia",
|
||||
"PatreonSeenMedia",
|
||||
"SubscribeStarFailedMedia",
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""DiscordFailedMedia — per-source dead-letter ledger of Discord files that
|
||||
keep failing to download or validate.
|
||||
|
||||
Mirror of SubscribeStarFailedMedia. After `attempts` reaches the dead-letter
|
||||
threshold a routine walk skips the file (recovery still retries it); a later
|
||||
clean download clears the row. `filehash` is the seen-ledger's key.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from sqlalchemy.types import DateTime
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class DiscordFailedMedia(Base):
|
||||
__tablename__ = "discord_failed_media"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("source_id", "filehash", name="uq_discord_failed_media_source_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
first_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
last_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
@@ -0,0 +1,34 @@
|
||||
"""DiscordSeenMedia — per-source ledger of Discord files already downloaded.
|
||||
|
||||
Mirror of SubscribeStarSeenMedia for the native Discord ingester (milestone
|
||||
428). `filehash` holds the ingester's per-file key, `<message_id>:<media_id>`:
|
||||
the attachment id, or for an embed a hash of its URL path. Not the file's
|
||||
position in the message — an edit that removes a file renumbers the rest
|
||||
(see `discord_client.MediaItem`). The message record's own gate is the
|
||||
synthetic `message:<id>` key in the same column.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from sqlalchemy.types import DateTime
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class DiscordSeenMedia(Base):
|
||||
__tablename__ = "discord_seen_media"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
seen_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
Reference in New Issue
Block a user