feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s

Discord was the last focus platform still on gallery-dl. This adds the native
path, mirrored from gallery-dl 1.32.13's discord extractor:

- discord_client: API v10 with the user token and gallery-dl's request
  profile (dated Firefox UA, Referer). Walks a server, category, forum,
  channel or thread in gallery-dl's order and pages each channel newest-first.
  Files are attachments, then embeds, then forwards, numbered across the
  message. The resume cursor is <channel>:<before>. Text-only messages are
  not posts, since gallery-dl never made them.
- discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans
  names on Linux (only `/` and control characters change), so existing files
  are skipped_disk rather than fetched again. Sidecars carry identity only.
  The message record keeps gallery-dl's keys, so parse_sidecar,
  derive_post_url and the drop grouping read it unchanged.
- The ledger keys on the attachment id (or a hash of an embed's URL path),
  not the file's position, which an edit can renumber. Migration 0111.
- DiscordIngester: token auth, body canary off (files-only drops are
  normal). Registered as native, verified by token, and serialised
  per-platform, since every source shares one user token.
- ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out
  on a multi-channel source now ends the quiet channel, not the whole walk.
  Clients without the seam behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 19:42:00 -04:00
co-authored by Claude Opus 5.5
parent 058fa85606
commit 84e5448941
15 changed files with 1579 additions and 15 deletions
+58
View File
@@ -319,6 +319,64 @@ async def test_tick_early_out_after_threshold(source_id, sync_engine, tmp_path):
assert client.consumed_posts == 2
class _FeedsClient(_FakeClient):
"""A source that is several feeds walked in turn (a Discord server's
channels), with the optional `skip_feed` seam (#4413). `feeds` is a list of
`pages` lists, one per feed."""
def __init__(self, feeds):
super().__init__([page for pages in feeds for page in pages])
self._feeds = feeds
self._skip = False
self.skips = 0
def skip_feed(self):
self._skip = True
self.skips += 1
def iter_posts(self, campaign_id, cursor=None):
for pages in self._feeds:
self._skip = False
self._pages = pages
for item in super().iter_posts(campaign_id, cursor):
yield item
if self._skip:
break
def _seed_seen_media(sync_engine, source_id, media):
factory = sessionmaker(sync_engine, expire_on_commit=False)
with factory() as s:
for m in media:
s.add(PatreonSeenMedia(source_id=source_id, filehash=_ledger_key(m), post_id=m.post_id))
s.commit()
@pytest.mark.asyncio
async def test_a_quiet_feed_ends_itself_not_the_walk(source_id, sync_engine, tmp_path):
"""#4413: a Discord server's first channel is all seen; the tick must still
reach the second channel's new file, instead of stopping at the first."""
quiet = [_media(f"a{i}", 1) for i in range(1, 5)]
_seed_seen_media(sync_engine, source_id, quiet)
fresh = _media("b1", 1)
client = _FeedsClient([
[(None, [(m.post_id, [m]) for m in quiet])],
[(None, [("b1", [fresh])])],
])
downloader = _FakeDownloader(tmp_path)
result = _ingester(sync_engine, tmp_path, client, downloader).run(
source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="tick", seen_threshold=2,
)
assert result.success is True
assert client.skips == 1
# The quiet feed stopped after its 2nd seen post; the next feed was walked.
assert client.consumed_posts == 3
assert downloader.download_calls == 1
assert "1 feed(s) caught up" in result.stdout
assert "reached end" not in result.stdout
# --- backfill -------------------------------------------------------------