feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
a copied platform list ('patreon', 'subscribestar') that went stale when
Discord moved over. Sources now carry `native_ingester` from the backend's
own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
backfill's next chunk: each one waited for the source's regular interval,
so an armed backfill sat idle until the next check (8h at the default) and
a five-chunk walk took most of two days. The in-flight guard and the
platform lock keep one chunk at a time. A failing source falls back to its
backoff, and a stalled or out-of-budget walk stops being due. It also runs
when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
defaults, sidecar-mirroring postprocessor and token injection are gone.
The naming test moves to the native downloader and still renders against
the real gallery-dl sidecar fixture. That is the guard that the files
gallery-dl wrote are found on disk rather than fetched again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
"""Discord file naming, checked against the metadata gallery-dl really emitted.
|
||||
|
||||
Discord moved from gallery-dl to the native downloader in milestone 428. Every
|
||||
file gallery-dl wrote is on disk under gallery-dl's naming, and the native
|
||||
downloader finds it there — skips it instead of fetching it again — only if it
|
||||
builds the SAME name. This file used to pin gallery-dl's patterns (#3999: they
|
||||
named keys the extractor never set, and every file landed as
|
||||
`None/<date>_None_<name>`); it now pins the native downloader to what those
|
||||
patterns produced from the same real metadata.
|
||||
|
||||
The fixture keeps the key set and value types of a real attachment sidecar
|
||||
from the operator's instance (2026-09-13), with every value invented.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from backend.app.services.discord_client import DiscordClient
|
||||
from backend.app.services.discord_downloader import channel_dir, media_stem
|
||||
from backend.app.utils.sidecar import find_sidecar
|
||||
|
||||
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
|
||||
|
||||
|
||||
def _message_from_the_fixture():
|
||||
"""The API message that produced the fixture's sidecar."""
|
||||
gdl = json.loads(_FIXTURE.read_text())
|
||||
date, time = gdl["date"].split(" ")
|
||||
return {
|
||||
"id": gdl["message_id"],
|
||||
"type": 0,
|
||||
"timestamp": f"{date}T{time}.000000+00:00",
|
||||
"content": gdl["message"],
|
||||
"attachments": [{"id": "500000000000000005", "url": gdl["url"]}],
|
||||
"embeds": [],
|
||||
"_meta": {"channel": gdl["channel"], "channel_id": gdl["channel_id"],
|
||||
"server": gdl["server"], "server_id": gdl["server_id"]},
|
||||
}
|
||||
|
||||
|
||||
def test_the_native_name_is_the_name_gallery_dl_wrote(tmp_path):
|
||||
"""gallery-dl's `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`
|
||||
in directory `{channel}`, rendered from the fixture."""
|
||||
msg = _message_from_the_fixture()
|
||||
[media] = DiscordClient.extract_media(msg)
|
||||
assert media_stem(msg, media) + "." + media.extension == (
|
||||
"20240716_300000000000000003_01_image.png"
|
||||
)
|
||||
assert channel_dir(tmp_path, "a", msg) == tmp_path / "a" / "discord" / "nsfw-drops"
|
||||
|
||||
|
||||
def test_a_gallery_dl_sidecar_on_disk_still_pairs_with_its_file(tmp_path):
|
||||
"""Files gallery-dl wrote keep their full sidecars beside them; the native
|
||||
name must be the stem those sidecars were named after."""
|
||||
msg = _message_from_the_fixture()
|
||||
[media] = DiscordClient.extract_media(msg)
|
||||
stem = media_stem(msg, media)
|
||||
(tmp_path / f"{stem}.png").write_bytes(b"x")
|
||||
(tmp_path / f"{stem}.json").write_text(_FIXTURE.read_text())
|
||||
assert find_sidecar(tmp_path / f"{stem}.png") == tmp_path / f"{stem}.json"
|
||||
|
||||
|
||||
def test_two_files_with_the_same_original_name_get_distinct_names():
|
||||
"""`image.png` is Discord's default name; two messages must not collide."""
|
||||
one = _message_from_the_fixture()
|
||||
two = {**one, "id": "300000000000000099"}
|
||||
[m1] = DiscordClient.extract_media(one)
|
||||
[m2] = DiscordClient.extract_media(two)
|
||||
assert media_stem(one, m1) != media_stem(two, m2)
|
||||
Reference in New Issue
Block a user