feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

- Recover and Recapture show on every native source. The menu gated them on
  a copied platform list ('patreon', 'subscribestar') that went stale when
  Discord moved over. Sources now carry `native_ingester` from the backend's
  own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
  backfill's next chunk: each one waited for the source's regular interval,
  so an armed backfill sat idle until the next check (8h at the default) and
  a five-chunk walk took most of two days. The in-flight guard and the
  platform lock keep one chunk at a time. A failing source falls back to its
  backoff, and a stalled or out-of-budget walk stops being due. It also runs
  when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
  defaults, sidecar-mirroring postprocessor and token injection are gone.
  The naming test moves to the native downloader and still renders against
  the real gallery-dl sidecar fixture. That is the guard that the files
  gallery-dl wrote are found on disk rather than fetched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 23:14:48 -04:00
co-authored by Claude Opus 5.5
parent 84e5448941
commit e5bdcd2596
11 changed files with 208 additions and 202 deletions
+5 -3
View File
@@ -6,9 +6,11 @@ existing file on disk (`skipped_disk`) instead of fetching it again:
<images>/<artist>/discord/<channel>/<YYYYMMDD>_<message_id>_<NN>_<name>.<ext>
That is FC's gallery-dl config (`gallery_dl.DISCORD_DIRECTORY` /
`DISCORD_FILENAME`) under the per-source base directory
`<images>/<artist>/<platform>`. The name is cleaned the way gallery-dl cleans it
That is what FC's gallery-dl config produced (directory `{channel}`, filename
`{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, under the
per-source base directory `<images>/<artist>/<platform>`), retired from that
config once Discord moved here; tests/test_discord_naming.py pins the match
against a real gallery-dl sidecar. The name is cleaned the way gallery-dl cleans it
on Linux — `/` becomes `_` and control characters are removed, nothing else
(`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`,
whose Windows set would turn a `:` in a channel or file name into `_` and miss
+4 -73
View File
@@ -95,48 +95,6 @@ BACKFILL_CHUNK_SECONDS = 600
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
# --- Discord naming ---------------------------------------------------------
#
# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory
# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits
# for an attachment: `channel` is a plain STRING (the channel's name), the
# message is `message_id`, the attachment's position in it is `num`, and there
# is NO `id` key at all.
#
# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as
# "None", so every Discord download since the platform was added landed in a
# directory called `None` as `<date>_None_<original name>`. Worse, the sidecar was
# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_
# sidecar` can never pair with `<date>_None_<name>.png`, so no Discord file ever
# got a Post or a post date, and (b) collides: every `image.png` in a channel
# overwrote the same `image.json`, so the one sidecar that survived described
# whichever message happened to be written last.
#
# The fix names the sidecar EXACTLY like the media minus its extension, so
# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match
# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders
# these patterns against a sanitized copy of the real sidecar, so a key that
# does not exist fails CI instead of silently becoming "None".
DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}"
DISCORD_DIRECTORY = ["{channel}"]
def sidecar_name_for(media_pattern: str) -> str | None:
"""The metadata filename pattern that names a sidecar exactly like its media.
Returns None for a pattern that does not end in `.{extension}`, since then
there is no media stem to mirror and the caller must fall back.
"""
suffix = ".{extension}"
if not media_pattern.endswith(suffix):
return None
return media_pattern[: -len(suffix)] + ".json"
def metadata_postprocessor(filename: str) -> dict:
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
def archive_path(images_root: Path) -> Path:
"""gallery-dl's download archive: the record of what it has already fetched.
@@ -425,27 +383,16 @@ class GalleryDLService:
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = {
# subscribestar removed — native-ingester platform now (#71); pixiv
# removed likewise (#129); deviantart removed at #3069 as a dropped
# platform, not a migrated one. The remaining entries are the
# gallery-dl platforms not yet migrated.
# removed likewise (#129); discord likewise (milestone 428, whose
# downloader keeps this config's on-disk naming); deviantart removed at
# #3069 as a dropped platform, not a migrated one. HentaiFoundry is the
# one platform left here, by the operator's choice not to migrate it.
"hentaifoundry": {
"content_types": ["all"],
"directory": [],
"filename": "{category}_{index:>03}_{title[:50]}.{extension}",
"include": "all",
},
"discord": {
"content_types": ["all"],
"directory": DISCORD_DIRECTORY,
"filename": DISCORD_FILENAME,
# Overrides the global `{filename}.json` sidecar for this extractor
# only — see the Discord naming note above.
"postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))],
"embeds": "all",
"stickers": True,
"reactions": False,
"threads": True,
},
}
def __init__(
@@ -560,17 +507,6 @@ class GalleryDLService:
if source_config.filename_pattern:
platform_section["filename"] = source_config.filename_pattern
# A platform that names its sidecar after its media must keep doing so
# under a per-source filename override, or the pairing breaks exactly the
# way Discord's did. No metadata wanted means no platform postprocessor
# either — the global list was already dropped above.
if "postprocessors" in platform_section:
mirrored = sidecar_name_for(platform_section.get("filename") or "")
if not source_config.save_metadata or mirrored is None:
platform_section.pop("postprocessors")
else:
platform_section["postprocessors"] = [metadata_postprocessor(mirrored)]
platform_section["metadata"] = source_config.save_metadata
return config
@@ -818,9 +754,6 @@ class GalleryDLService:
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})
config["extractor"]["discord"]["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
@@ -1004,8 +937,6 @@ class GalleryDLService:
config = self._build_config_for_source(platform, source_config, artist_slug)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
+36 -1
View File
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
return active
def backfill_ready(source: Source) -> bool:
"""A deep walk the operator started, with budget left and no failure
backing it off — due NOW rather than at its next scheduled check.
A backfill runs one time-boxed chunk per download (plan #693), and nothing
queued the next chunk: each waited for the source's regular interval. At
the 8-hour default a freshly armed backfill sat untouched until the next
check (the operator armed one on 2026-09-25 and saw nothing happen) and a
five-chunk walk took most of two days. The tick's in-flight guard keeps
one chunk at a time per source and the platform lock one walk per
platform, so "due every tick" means "next chunk as soon as the last one
ends".
The failure gate is what keeps a broken source from retrying every
minute: any failed chunk raises `consecutive_failures`, which drops the
source back onto its backed-off interval. A chunk that fails to progress
twice marks the walk stalled (download_service), which ends it here too.
"""
co = source.config_overrides or {}
return (
co.get("_backfill_state") == "running"
and (source.backfill_runs_remaining or 0) > 0
and not (source.consecutive_failures or 0)
)
async def select_due_sources(session: AsyncSession) -> list[Source]:
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
cooldown is the preventive half of the burst-prevention pair (per-source
consecutive_failures backoff handles the offending source itself).
A running backfill (`backfill_ready`) is due on every tick, and whether
or not its artist is on auto-check — the operator started it by hand.
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
sources go first, then the longest-since-checked, so the most overdue
sources hit Celery's FIFO download queue first. Anti-starvation: if
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
.options(selectinload(Source.artist))
.join(Artist, Source.artist_id == Artist.id)
.where(Source.enabled.is_(True))
.where(Artist.auto_check.is_(True))
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
)).scalars().all()
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
for s in rows:
if s.platform in cooldowns:
continue
if backfill_ready(s):
due.append(s)
continue
if not s.artist.auto_check:
continue
interval = compute_effective_interval(s, s.artist, settings)
if s.last_checked_at is None:
due.append(s)
@@ -161,6 +194,8 @@ def compute_next_check_at(
source: Source, artist: Artist, settings: ImportSettings,
) -> datetime | None:
"""Return the projected datetime of the next check, or None if never checked."""
if backfill_ready(source):
return datetime.now(UTC)
if source.last_checked_at is None:
return None
interval = compute_effective_interval(source, artist, settings)
+8 -2
View File
@@ -18,6 +18,7 @@ from ..models import (
Source,
)
from .db_helpers import failing_sources_clause
from .download_backends import uses_native_ingester
from .gallery_dl import ErrorType
from .membership_reconcile import KEPT_KEY, STOPPED_KEY
from .membership_roster import gated_reasons_for_sources
@@ -125,6 +126,11 @@ class SourceRecord:
"backfill_posts": self.backfill_posts,
"tier_gated_count": self.tier_gated_count,
"gated_reason": self.gated_reason,
# Recover / recapture exist only on the native ingester. Sent so the
# UI asks the backend's own predicate instead of keeping a copy of
# the platform list — the copy said "patreon, subscribestar" for a
# day after Discord went native (milestone 428).
"native_ingester": uses_native_ingester(self.platform),
}
@@ -551,8 +557,8 @@ class SourceService:
whole source); the two flags are mutually exclusive, so arming recapture
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
fresh from the top. The flag is cleared on completion (download_service)
and on stop. Recapture is Patreon-only (the native ingester's post-record
capture); inert elsewhere. The UI gates the action to Patreon sources."""
and on stop. Recapture needs the native ingester's post-record capture,
so the UI offers it on native sources only (`native_ingester`)."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
@@ -76,10 +76,10 @@ const running = computed(() => props.source.backfill_state === 'running')
const recovering = computed(() => !!props.source.backfill_bypass_seen)
const recapturing = computed(() => !!props.source.backfill_recapture)
// Recover / recapture are native-ingester features (ledger-bypass re-walk and
// post-text re-grab), available to every native platform — not just Patreon.
// Mirrors backend download_backends.NATIVE_INGESTER_PLATFORMS.
const NATIVE_PLATFORMS = ['patreon', 'subscribestar']
const isNative = computed(() => NATIVE_PLATFORMS.includes(props.source.platform))
// post-text re-grab), available on every native platform. The backend says
// which those are (`native_ingester`); a copied list here went stale when
// Discord moved over.
const isNative = computed(() => !!props.source.native_ingester)
</script>
<style scoped>
@@ -679,7 +679,7 @@ async function onRecover(source) {
}
}
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
// external links and localize on-disk inline images, without re-downloading
// media. Reuses the backfill lifecycle/badge; stop via the same Stop control.
async function onRecapture(source) {
+1 -1
View File
@@ -140,7 +140,7 @@ export const useSourcesStore = defineStore('sources', () => {
_patchSource(body)
return body
}
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
// external links and localize on-disk inline images, WITHOUT re-downloading
// media. Shares the backfill lifecycle/badge; stop via stopBackfill.
async function recaptureSource(id, artistIdHint = null) {
+69
View File
@@ -0,0 +1,69 @@
"""Discord file naming, checked against the metadata gallery-dl really emitted.
Discord moved from gallery-dl to the native downloader in milestone 428. Every
file gallery-dl wrote is on disk under gallery-dl's naming, and the native
downloader finds it there — skips it instead of fetching it again — only if it
builds the SAME name. This file used to pin gallery-dl's patterns (#3999: they
named keys the extractor never set, and every file landed as
`None/<date>_None_<name>`); it now pins the native downloader to what those
patterns produced from the same real metadata.
The fixture keeps the key set and value types of a real attachment sidecar
from the operator's instance (2026-09-13), with every value invented.
"""
import json
from pathlib import Path
from backend.app.services.discord_client import DiscordClient
from backend.app.services.discord_downloader import channel_dir, media_stem
from backend.app.utils.sidecar import find_sidecar
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
def _message_from_the_fixture():
"""The API message that produced the fixture's sidecar."""
gdl = json.loads(_FIXTURE.read_text())
date, time = gdl["date"].split(" ")
return {
"id": gdl["message_id"],
"type": 0,
"timestamp": f"{date}T{time}.000000+00:00",
"content": gdl["message"],
"attachments": [{"id": "500000000000000005", "url": gdl["url"]}],
"embeds": [],
"_meta": {"channel": gdl["channel"], "channel_id": gdl["channel_id"],
"server": gdl["server"], "server_id": gdl["server_id"]},
}
def test_the_native_name_is_the_name_gallery_dl_wrote(tmp_path):
"""gallery-dl's `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`
in directory `{channel}`, rendered from the fixture."""
msg = _message_from_the_fixture()
[media] = DiscordClient.extract_media(msg)
assert media_stem(msg, media) + "." + media.extension == (
"20240716_300000000000000003_01_image.png"
)
assert channel_dir(tmp_path, "a", msg) == tmp_path / "a" / "discord" / "nsfw-drops"
def test_a_gallery_dl_sidecar_on_disk_still_pairs_with_its_file(tmp_path):
"""Files gallery-dl wrote keep their full sidecars beside them; the native
name must be the stem those sidecars were named after."""
msg = _message_from_the_fixture()
[media] = DiscordClient.extract_media(msg)
stem = media_stem(msg, media)
(tmp_path / f"{stem}.png").write_bytes(b"x")
(tmp_path / f"{stem}.json").write_text(_FIXTURE.read_text())
assert find_sidecar(tmp_path / f"{stem}.png") == tmp_path / f"{stem}.json"
def test_two_files_with_the_same_original_name_get_distinct_names():
"""`image.png` is Discord's default name; two messages must not collide."""
one = _message_from_the_fixture()
two = {**one, "id": "300000000000000099"}
[m1] = DiscordClient.extract_media(one)
[m2] = DiscordClient.extract_media(two)
assert media_stem(one, m1) != media_stem(two, m2)
-117
View File
@@ -1,117 +0,0 @@
"""gallery-dl naming patterns, rendered against the metadata gallery-dl really emits.
gallery-dl does not fail on a format field that names a key the extractor never
sets. It renders "None" and carries on. That is how every Discord download
landed in a `None/` directory as `<date>_None_<name>`: the patterns asked for
`{channel[name]}` and `{id}`, and the real metadata has a string `channel` and
`message_id`, with no `id` at all. The mismatch also broke sidecar pairing, so
no Discord file ever got a Post or a post date.
The fixture keeps the key set and value TYPES of a real attachment sidecar from
the operator's instance (2026-09-13), with every value invented. Rendering
through Python's own formatter raises on a missing key or a subscript into a
string, which is the loud failure gallery-dl does not give.
"""
import json
import string
from datetime import datetime
from pathlib import Path
import pytest
from backend.app.services.gallery_dl import (
DISCORD_DIRECTORY,
DISCORD_FILENAME,
GalleryDLService,
SourceConfig,
sidecar_name_for,
)
from backend.app.utils.sidecar import find_sidecar
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
@pytest.fixture
def kwdict():
data = json.loads(_FIXTURE.read_text())
# gallery-dl hands the formatter a datetime; the JSON sidecar stores it as text.
data["date"] = datetime.strptime(data["date"], "%Y-%m-%d %H:%M:%S")
return data
def _render(pattern, kwdict):
return string.Formatter().vformat(pattern, (), kwdict)
def test_the_discord_filename_renders_from_real_keys(kwdict):
name = _render(DISCORD_FILENAME, kwdict)
assert "None" not in name
assert name == "20240716_300000000000000003_01_image.png"
def test_the_discord_directory_renders_from_real_keys(kwdict):
assert [_render(p, kwdict) for p in DISCORD_DIRECTORY] == ["nsfw-drops"]
@pytest.mark.parametrize(
"broken", ["{channel[name]}", "{date:%Y%m%d}_{id}_{filename}.{extension}"],
)
def test_the_patterns_that_shipped_would_have_failed_here(kwdict, broken):
"""Positive control: the two patterns that produced `None/..._None_...`
must fail this renderer, or the tests above prove nothing."""
with pytest.raises((KeyError, TypeError)):
_render(broken, kwdict)
def test_the_sidecar_is_named_exactly_like_its_media(kwdict, tmp_path):
"""So `find_sidecar` pairs them on its first candidate, and two attachments
that share an original name (`image.png`) can never share a sidecar."""
media = tmp_path / _render(DISCORD_FILENAME, kwdict)
sidecar = tmp_path / _render(sidecar_name_for(DISCORD_FILENAME), kwdict)
media.write_bytes(b"x")
sidecar.write_text("{}")
assert find_sidecar(media) == sidecar
def test_two_attachments_with_the_same_original_name_get_distinct_sidecars(kwdict):
other = {**kwdict, "message_id": "300000000000000099"}
pattern = sidecar_name_for(DISCORD_FILENAME)
assert _render(pattern, kwdict) != _render(pattern, other)
def test_sidecar_name_for_needs_an_extension_suffix():
assert sidecar_name_for("{a}_{b}.{extension}") == "{a}_{b}.json"
assert sidecar_name_for("{a}_{b}") is None
# --- the config gallery-dl is actually given -----------------------------------
@pytest.fixture
def gdl(tmp_path):
return GalleryDLService(images_root=tmp_path / "images", validate_files=False)
def _discord_section(gdl, **overrides):
cfg = gdl._build_config_for_source(
platform="discord", source_config=SourceConfig(**overrides), artist_slug="a",
)
return cfg["extractor"]["discord"]
def test_discord_config_carries_the_mirrored_sidecar(gdl):
section = _discord_section(gdl)
assert section["filename"] == DISCORD_FILENAME
assert section["directory"] == DISCORD_DIRECTORY
assert section["postprocessors"][0]["filename"] == sidecar_name_for(DISCORD_FILENAME)
def test_a_filename_override_keeps_the_sidecar_mirrored(gdl):
section = _discord_section(gdl, filename_pattern="{message_id}_{num}.{extension}")
assert section["postprocessors"][0]["filename"] == "{message_id}_{num}.json"
def test_no_metadata_means_no_discord_postprocessor(gdl):
section = _discord_section(gdl, save_metadata=False)
assert "postprocessors" not in section
+62
View File
@@ -11,6 +11,7 @@ import pytest
from backend.app.models import Artist, ImportSettings, Source
from backend.app.services.scheduler_service import (
backfill_ready,
compute_effective_interval,
scheduler_status,
select_due_sources,
@@ -150,6 +151,67 @@ async def test_select_includes_past_due(db):
assert any(s.url == "https://sel-past" for s in due)
# --- a running backfill is due every tick (2026-09-25) ---------------------
def _backfilling(runs=5, failures=0, state="running"):
src = _src(failures=failures)
src.config_overrides = {"_backfill_state": state}
src.backfill_runs_remaining = runs
return src
def test_a_running_backfill_is_ready():
assert backfill_ready(_backfilling()) is True
def test_a_backfill_out_of_budget_or_not_running_is_not_ready():
assert backfill_ready(_backfilling(runs=0)) is False
assert backfill_ready(_backfilling(state="stalled")) is False
assert backfill_ready(_src()) is False
def test_a_failing_backfill_falls_back_to_its_backoff():
"""Otherwise a source whose chunks keep erroring would retry every minute."""
assert backfill_ready(_backfilling(failures=1)) is False
@pytest.mark.asyncio
async def test_select_runs_a_backfill_now_not_at_its_next_check(db):
"""Checked a minute ago on an hourly interval: not due, unless backfilling —
the next chunk should not wait an interval (the 2026-09-25 armed-and-idle
backfill)."""
artist = await _seed_artist(db, interval=3600, name="bf-now")
for url, overrides, runs in (
("https://bf-running", {"_backfill_state": "running"}, 5),
("https://bf-idle", {}, 0),
):
db.add(Source(
artist_id=artist.id, platform="patreon", url=url, enabled=True,
consecutive_failures=0, config_overrides=overrides,
backfill_runs_remaining=runs,
last_checked_at=datetime.now(UTC) - timedelta(seconds=60),
))
await db.commit()
urls = {s.url for s in await select_due_sources(db)}
assert "https://bf-running" in urls
assert "https://bf-idle" not in urls
@pytest.mark.asyncio
async def test_a_backfill_runs_even_when_its_artist_is_not_auto_checked(db):
"""The operator started it by hand; auto-check governs the schedule only."""
artist = await _seed_artist(db, auto=False, name="bf-noauto")
db.add(Source(
artist_id=artist.id, platform="patreon", url="https://bf-noauto",
enabled=True, consecutive_failures=0,
config_overrides={"_backfill_state": "running"}, backfill_runs_remaining=5,
last_checked_at=datetime.now(UTC),
))
await db.commit()
assert any(s.url == "https://bf-noauto" for s in await select_due_sources(db))
# --- platform-rate-limit cooldown -----------------------------------------
+18
View File
@@ -34,6 +34,24 @@ async def test_known_platforms_are_the_supported_four(db):
assert "pixiv" not in KNOWN_PLATFORMS
@pytest.mark.asyncio
async def test_a_source_says_whether_it_is_on_the_native_ingester(db):
"""The UI offers recover / recapture from this flag rather than a copied
platform list — the copy went stale when Discord moved (milestone 428)."""
artist = await _artist(db, name="Natty")
svc = SourceService(db)
discord = await svc.create(
artist_id=artist.id, platform="discord",
url="https://discord.com/channels/1/2",
)
hf = await svc.create(
artist_id=artist.id, platform="hentaifoundry",
url="https://www.hentai-foundry.com/user/Natty",
)
assert discord.to_dict()["native_ingester"] is True
assert hf.to_dict()["native_ingester"] is False
@pytest.mark.asyncio
async def test_create_flips_is_subscription_on_first_source(db):
artist = await _artist(db)