bvandeusenandClaude Opus 5.5
e5bdcd2596
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
a copied platform list ('patreon', 'subscribestar') that went stale when
Discord moved over. Sources now carry `native_ingester` from the backend's
own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
backfill's next chunk: each one waited for the source's regular interval,
so an armed backfill sat idle until the next check (8h at the default) and
a five-chunk walk took most of two days. The in-flight guard and the
platform lock keep one chunk at a time. A failing source falls back to its
backoff, and a stalled or out-of-budget walk stops being due. It also runs
when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
defaults, sidecar-mirroring postprocessor and token injection are gone.
The naming test moves to the native downloader and still renders against
the real gallery-dl sidecar fixture. That is the guard that the files
gallery-dl wrote are found on disk rather than fetched again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
70 lines
3.0 KiB
Python
70 lines
3.0 KiB
Python
"""Discord file naming, checked against the metadata gallery-dl really emitted.
|
|
|
|
Discord moved from gallery-dl to the native downloader in milestone 428. Every
|
|
file gallery-dl wrote is on disk under gallery-dl's naming, and the native
|
|
downloader finds it there — skips it instead of fetching it again — only if it
|
|
builds the SAME name. This file used to pin gallery-dl's patterns (#3999: they
|
|
named keys the extractor never set, and every file landed as
|
|
`None/<date>_None_<name>`); it now pins the native downloader to what those
|
|
patterns produced from the same real metadata.
|
|
|
|
The fixture keeps the key set and value types of a real attachment sidecar
|
|
from the operator's instance (2026-09-13), with every value invented.
|
|
"""
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
from backend.app.services.discord_client import DiscordClient
|
|
from backend.app.services.discord_downloader import channel_dir, media_stem
|
|
from backend.app.utils.sidecar import find_sidecar
|
|
|
|
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
|
|
|
|
|
|
def _message_from_the_fixture():
|
|
"""The API message that produced the fixture's sidecar."""
|
|
gdl = json.loads(_FIXTURE.read_text())
|
|
date, time = gdl["date"].split(" ")
|
|
return {
|
|
"id": gdl["message_id"],
|
|
"type": 0,
|
|
"timestamp": f"{date}T{time}.000000+00:00",
|
|
"content": gdl["message"],
|
|
"attachments": [{"id": "500000000000000005", "url": gdl["url"]}],
|
|
"embeds": [],
|
|
"_meta": {"channel": gdl["channel"], "channel_id": gdl["channel_id"],
|
|
"server": gdl["server"], "server_id": gdl["server_id"]},
|
|
}
|
|
|
|
|
|
def test_the_native_name_is_the_name_gallery_dl_wrote(tmp_path):
|
|
"""gallery-dl's `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`
|
|
in directory `{channel}`, rendered from the fixture."""
|
|
msg = _message_from_the_fixture()
|
|
[media] = DiscordClient.extract_media(msg)
|
|
assert media_stem(msg, media) + "." + media.extension == (
|
|
"20240716_300000000000000003_01_image.png"
|
|
)
|
|
assert channel_dir(tmp_path, "a", msg) == tmp_path / "a" / "discord" / "nsfw-drops"
|
|
|
|
|
|
def test_a_gallery_dl_sidecar_on_disk_still_pairs_with_its_file(tmp_path):
|
|
"""Files gallery-dl wrote keep their full sidecars beside them; the native
|
|
name must be the stem those sidecars were named after."""
|
|
msg = _message_from_the_fixture()
|
|
[media] = DiscordClient.extract_media(msg)
|
|
stem = media_stem(msg, media)
|
|
(tmp_path / f"{stem}.png").write_bytes(b"x")
|
|
(tmp_path / f"{stem}.json").write_text(_FIXTURE.read_text())
|
|
assert find_sidecar(tmp_path / f"{stem}.png") == tmp_path / f"{stem}.json"
|
|
|
|
|
|
def test_two_files_with_the_same_original_name_get_distinct_names():
|
|
"""`image.png` is Discord's default name; two messages must not collide."""
|
|
one = _message_from_the_fixture()
|
|
two = {**one, "id": "300000000000000099"}
|
|
[m1] = DiscordClient.extract_media(one)
|
|
[m2] = DiscordClient.extract_media(two)
|
|
assert media_stem(one, m1) != media_stem(two, m2)
|