feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m27s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -264,6 +264,14 @@ class Ingester:
|
||||
# operator-driven "re-read every body" pass; a horizon there would be a
|
||||
# third overlapping answer to a question that has two.
|
||||
post_meta = getattr(self.client, "post_meta", None)
|
||||
# #4413: optional client seam for a source that is several feeds walked
|
||||
# one after another (a Discord server: every channel and thread). The
|
||||
# tick early-out means "this feed has nothing new", and without the
|
||||
# seam it ends the WHOLE walk — so the first quiet channel would hide
|
||||
# every channel after it. With it, the early-out asks the client to
|
||||
# move on and the walk continues. Absent → the early-out ends the walk,
|
||||
# exactly as before (Patreon and SubscribeStar are one feed each).
|
||||
skip_feed = getattr(self.client, "skip_feed", None)
|
||||
horizon: datetime | None = None
|
||||
if mode == "tick" and revisit_days > 0 and post_meta is not None:
|
||||
horizon = datetime.now(UTC) - timedelta(days=revisit_days)
|
||||
@@ -312,6 +320,7 @@ class Ingester:
|
||||
reached_bottom = False
|
||||
budget_hit = False
|
||||
early_out = False
|
||||
feeds_caught_up = 0 # #4413: feeds a tick left early via skip_feed
|
||||
stopped = False # plan #708 B4: operator hit Stop mid-walk
|
||||
cancel_armed = False # latched once we observe a live "running" state
|
||||
|
||||
@@ -674,9 +683,15 @@ class Ingester:
|
||||
})
|
||||
|
||||
if early_out:
|
||||
break
|
||||
if skip_feed is None:
|
||||
break
|
||||
skip_feed()
|
||||
feeds_caught_up += 1
|
||||
early_out = False
|
||||
consecutive_seen = 0
|
||||
else:
|
||||
reached_bottom = True
|
||||
# A walk that left feeds early did not read to their ends.
|
||||
reached_bottom = not feeds_caught_up
|
||||
except self._error_base as exc:
|
||||
# The platform's client-error base — _failure_result (adapter)
|
||||
# maps it to a typed error.
|
||||
@@ -724,6 +739,7 @@ class Ingester:
|
||||
f", {revisited} post(s) updated ({revisit_downloads} new file(s))"
|
||||
if revisited else ""
|
||||
)
|
||||
+ (f", {feeds_caught_up} feed(s) caught up" if feeds_caught_up else "")
|
||||
+ (", reached end" if reached_bottom else "")
|
||||
+ (", time-boxed" if budget_hit else "")
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user