fix: a Discord source takes only messages with an image attached, from the posters it names, and knows its channel's name (#4481)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / extension-test (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 59s
CI and images / promote (push) Successful in 2s
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / extension-test (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 59s
CI and images / promote (push) Successful in 2s
- A message with no image or video attachment is chat: extract_media returns nothing for it, so nothing downloads and no post record is written. A message that passes keeps every file, numbered as gallery-dl numbers them. - `discord_authors` in a source's config limits the walk to those posters (id, username or display name); the edit dialog has a field for it and no longer drops config keys it has no field for. - source.display_name (migration 0115), refreshed by every Discord walk, shows on Subscriptions in place of the two-id URL; a Discord post card names its channel from the record's `channel`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -13,6 +13,11 @@ Two things differ from the cookie platforms:
|
||||
something; a Discord drop is routinely files and nothing else, so on
|
||||
Discord that is an ordinary backfill, not a broken parser.
|
||||
|
||||
And two things only a Discord source has: `discord_authors` in its
|
||||
config_overrides limits it to the creator's own messages (a server's other
|
||||
members post pictures too), and it learns its own name — the server and
|
||||
channel it walks — into `source.display_name`.
|
||||
|
||||
`campaign_id` is the source URL (a server, channel, thread or category link).
|
||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
||||
"""
|
||||
@@ -20,15 +25,23 @@ FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from ..models import DiscordFailedMedia, DiscordSeenMedia
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from ..models import DiscordFailedMedia, DiscordSeenMedia, Source
|
||||
from .discord_client import DiscordAPIError, DiscordClient, MediaItem
|
||||
from .discord_downloader import DiscordDownloader
|
||||
from .ingest_core import Ingester
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_LEDGER_KEY_MAX = 128
|
||||
# `config_overrides` key: the people whose messages this source takes (user
|
||||
# ids, usernames or display names). Absent or empty takes everyone's.
|
||||
AUTHORS_KEY = "discord_authors"
|
||||
|
||||
|
||||
def _ledger_key(media: MediaItem) -> str:
|
||||
@@ -74,6 +87,46 @@ class DiscordIngester(Ingester):
|
||||
body_canary=False,
|
||||
)
|
||||
|
||||
def run(self, **kwargs):
|
||||
"""The core walk, bracketed by the two things only a Discord source
|
||||
has: whose messages it takes, read before the walk, and the name of
|
||||
what it walks, written after it (the walk is what loads that name)."""
|
||||
source_id = kwargs["source_id"]
|
||||
self.client.only_from(self._source_authors(source_id))
|
||||
try:
|
||||
return super().run(**kwargs)
|
||||
finally:
|
||||
self._record_display_name(source_id, kwargs["campaign_id"])
|
||||
|
||||
def _source_authors(self, source_id: int) -> list:
|
||||
if self.session_factory is None:
|
||||
return []
|
||||
with self.session_factory() as session:
|
||||
overrides = session.execute(
|
||||
select(Source.config_overrides).where(Source.id == source_id)
|
||||
).scalar_one_or_none() or {}
|
||||
authors = overrides.get(AUTHORS_KEY) or []
|
||||
return authors if isinstance(authors, list) else []
|
||||
|
||||
def _record_display_name(self, source_id: int, url: str) -> None:
|
||||
"""Refreshed on every walk, never written once: a renamed channel
|
||||
should read as its new name. A name the walk couldn't read leaves the
|
||||
stored one alone, and a failure here never fails the walk."""
|
||||
label = self.client.source_label(url)
|
||||
if not label or self.session_factory is None:
|
||||
return
|
||||
try:
|
||||
with self.session_factory() as session:
|
||||
session.execute(
|
||||
update(Source)
|
||||
.where(Source.id == source_id)
|
||||
.where(Source.display_name.is_distinct_from(label))
|
||||
.values(display_name=label)
|
||||
)
|
||||
session.commit()
|
||||
except Exception as exc: # a name is decoration — never fail the walk
|
||||
log.warning("Discord: couldn't record source %s's name: %s", source_id, exc)
|
||||
|
||||
|
||||
async def verify_discord_credential(url: str, auth_token: str | None) -> tuple[bool | None, str]:
|
||||
"""The uniform `(ok, message)` probe: is the token valid, and can its
|
||||
|
||||
Reference in New Issue
Block a user