feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

- Recover and Recapture show on every native source. The menu gated them on
  a copied platform list ('patreon', 'subscribestar') that went stale when
  Discord moved over. Sources now carry `native_ingester` from the backend's
  own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
  backfill's next chunk: each one waited for the source's regular interval,
  so an armed backfill sat idle until the next check (8h at the default) and
  a five-chunk walk took most of two days. The in-flight guard and the
  platform lock keep one chunk at a time. A failing source falls back to its
  backoff, and a stalled or out-of-budget walk stops being due. It also runs
  when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
  defaults, sidecar-mirroring postprocessor and token injection are gone.
  The naming test moves to the native downloader and still renders against
  the real gallery-dl sidecar fixture. That is the guard that the files
  gallery-dl wrote are found on disk rather than fetched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 23:14:48 -04:00
co-authored by Claude Opus 5.5
parent 84e5448941
commit e5bdcd2596
11 changed files with 208 additions and 202 deletions
+4 -73
View File
@@ -95,48 +95,6 @@ BACKFILL_CHUNK_SECONDS = 600
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
# --- Discord naming ---------------------------------------------------------
#
# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory
# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits
# for an attachment: `channel` is a plain STRING (the channel's name), the
# message is `message_id`, the attachment's position in it is `num`, and there
# is NO `id` key at all.
#
# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as
# "None", so every Discord download since the platform was added landed in a
# directory called `None` as `<date>_None_<original name>`. Worse, the sidecar was
# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_
# sidecar` can never pair with `<date>_None_<name>.png`, so no Discord file ever
# got a Post or a post date, and (b) collides: every `image.png` in a channel
# overwrote the same `image.json`, so the one sidecar that survived described
# whichever message happened to be written last.
#
# The fix names the sidecar EXACTLY like the media minus its extension, so
# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match
# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders
# these patterns against a sanitized copy of the real sidecar, so a key that
# does not exist fails CI instead of silently becoming "None".
DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}"
DISCORD_DIRECTORY = ["{channel}"]
def sidecar_name_for(media_pattern: str) -> str | None:
"""The metadata filename pattern that names a sidecar exactly like its media.
Returns None for a pattern that does not end in `.{extension}`, since then
there is no media stem to mirror and the caller must fall back.
"""
suffix = ".{extension}"
if not media_pattern.endswith(suffix):
return None
return media_pattern[: -len(suffix)] + ".json"
def metadata_postprocessor(filename: str) -> dict:
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
def archive_path(images_root: Path) -> Path:
"""gallery-dl's download archive: the record of what it has already fetched.
@@ -425,27 +383,16 @@ class GalleryDLService:
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = {
# subscribestar removed — native-ingester platform now (#71); pixiv
# removed likewise (#129); deviantart removed at #3069 as a dropped
# platform, not a migrated one. The remaining entries are the
# gallery-dl platforms not yet migrated.
# removed likewise (#129); discord likewise (milestone 428, whose
# downloader keeps this config's on-disk naming); deviantart removed at
# #3069 as a dropped platform, not a migrated one. HentaiFoundry is the
# one platform left here, by the operator's choice not to migrate it.
"hentaifoundry": {
"content_types": ["all"],
"directory": [],
"filename": "{category}_{index:>03}_{title[:50]}.{extension}",
"include": "all",
},
"discord": {
"content_types": ["all"],
"directory": DISCORD_DIRECTORY,
"filename": DISCORD_FILENAME,
# Overrides the global `{filename}.json` sidecar for this extractor
# only — see the Discord naming note above.
"postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))],
"embeds": "all",
"stickers": True,
"reactions": False,
"threads": True,
},
}
def __init__(
@@ -560,17 +507,6 @@ class GalleryDLService:
if source_config.filename_pattern:
platform_section["filename"] = source_config.filename_pattern
# A platform that names its sidecar after its media must keep doing so
# under a per-source filename override, or the pairing breaks exactly the
# way Discord's did. No metadata wanted means no platform postprocessor
# either — the global list was already dropped above.
if "postprocessors" in platform_section:
mirrored = sidecar_name_for(platform_section.get("filename") or "")
if not source_config.save_metadata or mirrored is None:
platform_section.pop("postprocessors")
else:
platform_section["postprocessors"] = [metadata_postprocessor(mirrored)]
platform_section["metadata"] = source_config.save_metadata
return config
@@ -818,9 +754,6 @@ class GalleryDLService:
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})
config["extractor"]["discord"]["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
@@ -1004,8 +937,6 @@ class GalleryDLService:
config = self._build_config_for_source(platform, source_config, artist_slug)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),