feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -319,6 +319,64 @@ async def test_tick_early_out_after_threshold(source_id, sync_engine, tmp_path):
|
||||
assert client.consumed_posts == 2
|
||||
|
||||
|
||||
class _FeedsClient(_FakeClient):
|
||||
"""A source that is several feeds walked in turn (a Discord server's
|
||||
channels), with the optional `skip_feed` seam (#4413). `feeds` is a list of
|
||||
`pages` lists, one per feed."""
|
||||
|
||||
def __init__(self, feeds):
|
||||
super().__init__([page for pages in feeds for page in pages])
|
||||
self._feeds = feeds
|
||||
self._skip = False
|
||||
self.skips = 0
|
||||
|
||||
def skip_feed(self):
|
||||
self._skip = True
|
||||
self.skips += 1
|
||||
|
||||
def iter_posts(self, campaign_id, cursor=None):
|
||||
for pages in self._feeds:
|
||||
self._skip = False
|
||||
self._pages = pages
|
||||
for item in super().iter_posts(campaign_id, cursor):
|
||||
yield item
|
||||
if self._skip:
|
||||
break
|
||||
|
||||
|
||||
def _seed_seen_media(sync_engine, source_id, media):
|
||||
factory = sessionmaker(sync_engine, expire_on_commit=False)
|
||||
with factory() as s:
|
||||
for m in media:
|
||||
s.add(PatreonSeenMedia(source_id=source_id, filehash=_ledger_key(m), post_id=m.post_id))
|
||||
s.commit()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_quiet_feed_ends_itself_not_the_walk(source_id, sync_engine, tmp_path):
|
||||
"""#4413: a Discord server's first channel is all seen; the tick must still
|
||||
reach the second channel's new file, instead of stopping at the first."""
|
||||
quiet = [_media(f"a{i}", 1) for i in range(1, 5)]
|
||||
_seed_seen_media(sync_engine, source_id, quiet)
|
||||
fresh = _media("b1", 1)
|
||||
client = _FeedsClient([
|
||||
[(None, [(m.post_id, [m]) for m in quiet])],
|
||||
[(None, [("b1", [fresh])])],
|
||||
])
|
||||
downloader = _FakeDownloader(tmp_path)
|
||||
result = _ingester(sync_engine, tmp_path, client, downloader).run(
|
||||
source_id=source_id, campaign_id="c1", artist_slug="ingest",
|
||||
url="https://patreon.com/ingest", mode="tick", seen_threshold=2,
|
||||
)
|
||||
assert result.success is True
|
||||
assert client.skips == 1
|
||||
# The quiet feed stopped after its 2nd seen post; the next feed was walked.
|
||||
assert client.consumed_posts == 3
|
||||
assert downloader.download_calls == 1
|
||||
assert "1 feed(s) caught up" in result.stdout
|
||||
assert "reached end" not in result.stdout
|
||||
|
||||
|
||||
# --- backfill -------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user