Files
FabledCurator/tests/test_gallery_dl_naming.py
T
bvandeusenandClaude Opus 5 dc840feec7
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 1m17s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m15s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m34s
fix: Discord downloads land as None/<date>_None_<name> with no post — name them from the keys gallery-dl really emits
Operator report: a Discord source showed the wrong post time. Listing the real downloads found every Discord folder on the instance (7 artist folders, ~1,600 media) in a directory named `None`, as `<date>_None_<original name>`, next to sidecars named `<original name>.json`.

The real sidecar settles why:
- gallery-dl's discord extractor emits `channel` as a plain string, the message as `message_id`, and the attachment position as `num`. It has no `id` key.
- The patterns asked for `{channel[name]}` and `{id}`. gallery-dl renders a missing field as "None" and carries on.
- The sidecar was named `{filename}.json`, the attachment's ORIGINAL name. find_sidecar never pairs that with `<date>_None_<name>.png`, so no Discord file ever got a Post or a post date. The card fell back to downloaded_at.
- Every `image.png` in a channel also overwrote the same `image.json`.

Fix (gallery_dl.py):
- The directory is `{channel}`.
- The filename is `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, unique per attachment.
- A discord-scoped metadata postprocessor names the sidecar exactly like the media minus its extension, so find_sidecar's first candidate matches. A per-source filename override re-derives the sidecar name. save_metadata=False drops it.

Guard (tests/test_gallery_dl_naming.py) renders the patterns through Python's formatter against a sanitized copy of the real sidecar (same keys and types, invented values). A missing key or a subscript into a string raises, which is the loud failure gallery-dl doesn't give. A positive control shows both shipped patterns fail it.

Existing broken downloads are NOT repaired by this. gallery-dl's archive already records them, so a re-run skips them, and their collided sidecars no longer describe them. That repair is a separate, destructive step for the operator to decide on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
2026-09-13 20:01:11 -04:00

118 lines
4.2 KiB
Python

"""gallery-dl naming patterns, rendered against the metadata gallery-dl really emits.
gallery-dl does not fail on a format field that names a key the extractor never
sets. It renders "None" and carries on. That is how every Discord download
landed in a `None/` directory as `<date>_None_<name>`: the patterns asked for
`{channel[name]}` and `{id}`, and the real metadata has a string `channel` and
`message_id`, with no `id` at all. The mismatch also broke sidecar pairing, so
no Discord file ever got a Post or a post date.
The fixture keeps the key set and value TYPES of a real attachment sidecar from
the operator's instance (2026-09-13), with every value invented. Rendering
through Python's own formatter raises on a missing key or a subscript into a
string, which is the loud failure gallery-dl does not give.
"""
import json
import string
from datetime import datetime
from pathlib import Path
import pytest
from backend.app.services.gallery_dl import (
DISCORD_DIRECTORY,
DISCORD_FILENAME,
GalleryDLService,
SourceConfig,
sidecar_name_for,
)
from backend.app.utils.sidecar import find_sidecar
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
@pytest.fixture
def kwdict():
data = json.loads(_FIXTURE.read_text())
# gallery-dl hands the formatter a datetime; the JSON sidecar stores it as text.
data["date"] = datetime.strptime(data["date"], "%Y-%m-%d %H:%M:%S")
return data
def _render(pattern, kwdict):
return string.Formatter().vformat(pattern, (), kwdict)
def test_the_discord_filename_renders_from_real_keys(kwdict):
name = _render(DISCORD_FILENAME, kwdict)
assert "None" not in name
assert name == "20240716_300000000000000003_01_image.png"
def test_the_discord_directory_renders_from_real_keys(kwdict):
assert [_render(p, kwdict) for p in DISCORD_DIRECTORY] == ["nsfw-drops"]
@pytest.mark.parametrize(
"broken", ["{channel[name]}", "{date:%Y%m%d}_{id}_{filename}.{extension}"],
)
def test_the_patterns_that_shipped_would_have_failed_here(kwdict, broken):
"""Positive control: the two patterns that produced `None/..._None_...`
must fail this renderer, or the tests above prove nothing."""
with pytest.raises((KeyError, TypeError)):
_render(broken, kwdict)
def test_the_sidecar_is_named_exactly_like_its_media(kwdict, tmp_path):
"""So `find_sidecar` pairs them on its first candidate, and two attachments
that share an original name (`image.png`) can never share a sidecar."""
media = tmp_path / _render(DISCORD_FILENAME, kwdict)
sidecar = tmp_path / _render(sidecar_name_for(DISCORD_FILENAME), kwdict)
media.write_bytes(b"x")
sidecar.write_text("{}")
assert find_sidecar(media) == sidecar
def test_two_attachments_with_the_same_original_name_get_distinct_sidecars(kwdict):
other = {**kwdict, "message_id": "300000000000000099"}
pattern = sidecar_name_for(DISCORD_FILENAME)
assert _render(pattern, kwdict) != _render(pattern, other)
def test_sidecar_name_for_needs_an_extension_suffix():
assert sidecar_name_for("{a}_{b}.{extension}") == "{a}_{b}.json"
assert sidecar_name_for("{a}_{b}") is None
# --- the config gallery-dl is actually given -----------------------------------
@pytest.fixture
def gdl(tmp_path):
return GalleryDLService(images_root=tmp_path / "images", validate_files=False)
def _discord_section(gdl, **overrides):
cfg = gdl._build_config_for_source(
platform="discord", source_config=SourceConfig(**overrides), artist_slug="a",
)
return cfg["extractor"]["discord"]
def test_discord_config_carries_the_mirrored_sidecar(gdl):
section = _discord_section(gdl)
assert section["filename"] == DISCORD_FILENAME
assert section["directory"] == DISCORD_DIRECTORY
assert section["postprocessors"][0]["filename"] == sidecar_name_for(DISCORD_FILENAME)
def test_a_filename_override_keeps_the_sidecar_mirrored(gdl):
section = _discord_section(gdl, filename_pattern="{message_id}_{num}.{extension}")
assert section["postprocessors"][0]["filename"] == "{message_id}_{num}.json"
def test_no_metadata_means_no_discord_postprocessor(gdl):
section = _discord_section(gdl, save_metadata=False)
assert "postprocessors" not in section