fix: Discord downloads land as None/<date>_None_<name> with no post — name them from the keys gallery-dl really emits
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 1m17s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m15s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m34s

Operator report: a Discord source showed the wrong post time. Listing the real downloads found every Discord folder on the instance (7 artist folders, ~1,600 media) in a directory named `None`, as `<date>_None_<original name>`, next to sidecars named `<original name>.json`.

The real sidecar settles why:
- gallery-dl's discord extractor emits `channel` as a plain string, the message as `message_id`, and the attachment position as `num`. It has no `id` key.
- The patterns asked for `{channel[name]}` and `{id}`. gallery-dl renders a missing field as "None" and carries on.
- The sidecar was named `{filename}.json`, the attachment's ORIGINAL name. find_sidecar never pairs that with `<date>_None_<name>.png`, so no Discord file ever got a Post or a post date. The card fell back to downloaded_at.
- Every `image.png` in a channel also overwrote the same `image.json`.

Fix (gallery_dl.py):
- The directory is `{channel}`.
- The filename is `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, unique per attachment.
- A discord-scoped metadata postprocessor names the sidecar exactly like the media minus its extension, so find_sidecar's first candidate matches. A per-source filename override re-derives the sidecar name. save_metadata=False drops it.

Guard (tests/test_gallery_dl_naming.py) renders the patterns through Python's formatter against a sanitized copy of the real sidecar (same keys and types, invented values). A missing key or a subscript into a string raises, which is the loud failure gallery-dl doesn't give. A positive control shows both shipped patterns fail it.

Existing broken downloads are NOT repaired by this. gallery-dl's archive already records them, so a re-run skips them, and their collided sidecars no longer describe them. That repair is a separate, destructive step for the operator to decide on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
This commit is contained in:
2026-09-13 20:01:11 -04:00
co-authored by Claude Opus 5
parent 98c3b74260
commit dc840feec7
3 changed files with 203 additions and 2 deletions
+58 -2
View File
@@ -94,6 +94,48 @@ BACKFILL_CHUNK_SECONDS = 600
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
# --- Discord naming ---------------------------------------------------------
#
# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory
# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits
# for an attachment: `channel` is a plain STRING (the channel's name), the
# message is `message_id`, the attachment's position in it is `num`, and there
# is NO `id` key at all.
#
# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as
# "None", so every Discord download since the platform was added landed in a
# directory called `None` as `<date>_None_<original name>`. Worse, the sidecar was
# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_
# sidecar` can never pair with `<date>_None_<name>.png`, so no Discord file ever
# got a Post or a post date, and (b) collides: every `image.png` in a channel
# overwrote the same `image.json`, so the one sidecar that survived described
# whichever message happened to be written last.
#
# The fix names the sidecar EXACTLY like the media minus its extension, so
# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match
# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders
# these patterns against a sanitized copy of the real sidecar, so a key that
# does not exist fails CI instead of silently becoming "None".
DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}"
DISCORD_DIRECTORY = ["{channel}"]
def sidecar_name_for(media_pattern: str) -> str | None:
"""The metadata filename pattern that names a sidecar exactly like its media.
Returns None for a pattern that does not end in `.{extension}`, since then
there is no media stem to mirror and the caller must fall back.
"""
suffix = ".{extension}"
if not media_pattern.endswith(suffix):
return None
return media_pattern[: -len(suffix)] + ".json"
def metadata_postprocessor(filename: str) -> dict:
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
@dataclass
class SourceConfig:
"""Per-source overrides loaded from Source.config_overrides JSON.
@@ -376,8 +418,11 @@ class GalleryDLService:
},
"discord": {
"content_types": ["all"],
"directory": ["{channel[name]}"],
"filename": "{date:%Y%m%d}_{id}_{filename}.{extension}",
"directory": DISCORD_DIRECTORY,
"filename": DISCORD_FILENAME,
# Overrides the global `{filename}.json` sidecar for this extractor
# only — see the Discord naming note above.
"postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))],
"embeds": "all",
"stickers": True,
"reactions": False,
@@ -497,6 +542,17 @@ class GalleryDLService:
if source_config.filename_pattern:
platform_section["filename"] = source_config.filename_pattern
# A platform that names its sidecar after its media must keep doing so
# under a per-source filename override, or the pairing breaks exactly the
# way Discord's did. No metadata wanted means no platform postprocessor
# either — the global list was already dropped above.
if "postprocessors" in platform_section:
mirrored = sidecar_name_for(platform_section.get("filename") or "")
if not source_config.save_metadata or mirrored is None:
platform_section.pop("postprocessors")
else:
platform_section["postprocessors"] = [metadata_postprocessor(mirrored)]
platform_section["metadata"] = source_config.save_metadata
return config