feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
a copied platform list ('patreon', 'subscribestar') that went stale when
Discord moved over. Sources now carry `native_ingester` from the backend's
own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
backfill's next chunk: each one waited for the source's regular interval,
so an armed backfill sat idle until the next check (8h at the default) and
a five-chunk walk took most of two days. The in-flight guard and the
platform lock keep one chunk at a time. A failing source falls back to its
backoff, and a stalled or out-of-budget walk stops being due. It also runs
when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
defaults, sidecar-mirroring postprocessor and token injection are gone.
The naming test moves to the native downloader and still renders against
the real gallery-dl sidecar fixture. That is the guard that the files
gallery-dl wrote are found on disk rather than fetched again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -6,9 +6,11 @@ existing file on disk (`skipped_disk`) instead of fetching it again:
|
|||||||
|
|
||||||
<images>/<artist>/discord/<channel>/<YYYYMMDD>_<message_id>_<NN>_<name>.<ext>
|
<images>/<artist>/discord/<channel>/<YYYYMMDD>_<message_id>_<NN>_<name>.<ext>
|
||||||
|
|
||||||
That is FC's gallery-dl config (`gallery_dl.DISCORD_DIRECTORY` /
|
That is what FC's gallery-dl config produced (directory `{channel}`, filename
|
||||||
`DISCORD_FILENAME`) under the per-source base directory
|
`{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, under the
|
||||||
`<images>/<artist>/<platform>`. The name is cleaned the way gallery-dl cleans it
|
per-source base directory `<images>/<artist>/<platform>`), retired from that
|
||||||
|
config once Discord moved here; tests/test_discord_naming.py pins the match
|
||||||
|
against a real gallery-dl sidecar. The name is cleaned the way gallery-dl cleans it
|
||||||
on Linux — `/` becomes `_` and control characters are removed, nothing else
|
on Linux — `/` becomes `_` and control characters are removed, nothing else
|
||||||
(`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`,
|
(`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`,
|
||||||
whose Windows set would turn a `:` in a channel or file name into `_` and miss
|
whose Windows set would turn a `:` in a channel or file name into `_` and miss
|
||||||
|
|||||||
@@ -95,48 +95,6 @@ BACKFILL_CHUNK_SECONDS = 600
|
|||||||
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
|
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
|
||||||
|
|
||||||
|
|
||||||
# --- Discord naming ---------------------------------------------------------
|
|
||||||
#
|
|
||||||
# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory
|
|
||||||
# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits
|
|
||||||
# for an attachment: `channel` is a plain STRING (the channel's name), the
|
|
||||||
# message is `message_id`, the attachment's position in it is `num`, and there
|
|
||||||
# is NO `id` key at all.
|
|
||||||
#
|
|
||||||
# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as
|
|
||||||
# "None", so every Discord download since the platform was added landed in a
|
|
||||||
# directory called `None` as `<date>_None_<original name>`. Worse, the sidecar was
|
|
||||||
# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_
|
|
||||||
# sidecar` can never pair with `<date>_None_<name>.png`, so no Discord file ever
|
|
||||||
# got a Post or a post date, and (b) collides: every `image.png` in a channel
|
|
||||||
# overwrote the same `image.json`, so the one sidecar that survived described
|
|
||||||
# whichever message happened to be written last.
|
|
||||||
#
|
|
||||||
# The fix names the sidecar EXACTLY like the media minus its extension, so
|
|
||||||
# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match
|
|
||||||
# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders
|
|
||||||
# these patterns against a sanitized copy of the real sidecar, so a key that
|
|
||||||
# does not exist fails CI instead of silently becoming "None".
|
|
||||||
DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}"
|
|
||||||
DISCORD_DIRECTORY = ["{channel}"]
|
|
||||||
|
|
||||||
|
|
||||||
def sidecar_name_for(media_pattern: str) -> str | None:
|
|
||||||
"""The metadata filename pattern that names a sidecar exactly like its media.
|
|
||||||
|
|
||||||
Returns None for a pattern that does not end in `.{extension}`, since then
|
|
||||||
there is no media stem to mirror and the caller must fall back.
|
|
||||||
"""
|
|
||||||
suffix = ".{extension}"
|
|
||||||
if not media_pattern.endswith(suffix):
|
|
||||||
return None
|
|
||||||
return media_pattern[: -len(suffix)] + ".json"
|
|
||||||
|
|
||||||
|
|
||||||
def metadata_postprocessor(filename: str) -> dict:
|
|
||||||
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
|
|
||||||
|
|
||||||
|
|
||||||
def archive_path(images_root: Path) -> Path:
|
def archive_path(images_root: Path) -> Path:
|
||||||
"""gallery-dl's download archive: the record of what it has already fetched.
|
"""gallery-dl's download archive: the record of what it has already fetched.
|
||||||
|
|
||||||
@@ -425,27 +383,16 @@ class GalleryDLService:
|
|||||||
# (services/patreon_ingester.py), not gallery-dl.
|
# (services/patreon_ingester.py), not gallery-dl.
|
||||||
PLATFORM_DEFAULTS = {
|
PLATFORM_DEFAULTS = {
|
||||||
# subscribestar removed — native-ingester platform now (#71); pixiv
|
# subscribestar removed — native-ingester platform now (#71); pixiv
|
||||||
# removed likewise (#129); deviantart removed at #3069 as a dropped
|
# removed likewise (#129); discord likewise (milestone 428, whose
|
||||||
# platform, not a migrated one. The remaining entries are the
|
# downloader keeps this config's on-disk naming); deviantart removed at
|
||||||
# gallery-dl platforms not yet migrated.
|
# #3069 as a dropped platform, not a migrated one. HentaiFoundry is the
|
||||||
|
# one platform left here, by the operator's choice not to migrate it.
|
||||||
"hentaifoundry": {
|
"hentaifoundry": {
|
||||||
"content_types": ["all"],
|
"content_types": ["all"],
|
||||||
"directory": [],
|
"directory": [],
|
||||||
"filename": "{category}_{index:>03}_{title[:50]}.{extension}",
|
"filename": "{category}_{index:>03}_{title[:50]}.{extension}",
|
||||||
"include": "all",
|
"include": "all",
|
||||||
},
|
},
|
||||||
"discord": {
|
|
||||||
"content_types": ["all"],
|
|
||||||
"directory": DISCORD_DIRECTORY,
|
|
||||||
"filename": DISCORD_FILENAME,
|
|
||||||
# Overrides the global `{filename}.json` sidecar for this extractor
|
|
||||||
# only — see the Discord naming note above.
|
|
||||||
"postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))],
|
|
||||||
"embeds": "all",
|
|
||||||
"stickers": True,
|
|
||||||
"reactions": False,
|
|
||||||
"threads": True,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -560,17 +507,6 @@ class GalleryDLService:
|
|||||||
if source_config.filename_pattern:
|
if source_config.filename_pattern:
|
||||||
platform_section["filename"] = source_config.filename_pattern
|
platform_section["filename"] = source_config.filename_pattern
|
||||||
|
|
||||||
# A platform that names its sidecar after its media must keep doing so
|
|
||||||
# under a per-source filename override, or the pairing breaks exactly the
|
|
||||||
# way Discord's did. No metadata wanted means no platform postprocessor
|
|
||||||
# either — the global list was already dropped above.
|
|
||||||
if "postprocessors" in platform_section:
|
|
||||||
mirrored = sidecar_name_for(platform_section.get("filename") or "")
|
|
||||||
if not source_config.save_metadata or mirrored is None:
|
|
||||||
platform_section.pop("postprocessors")
|
|
||||||
else:
|
|
||||||
platform_section["postprocessors"] = [metadata_postprocessor(mirrored)]
|
|
||||||
|
|
||||||
platform_section["metadata"] = source_config.save_metadata
|
platform_section["metadata"] = source_config.save_metadata
|
||||||
|
|
||||||
return config
|
return config
|
||||||
@@ -818,9 +754,6 @@ class GalleryDLService:
|
|||||||
|
|
||||||
if cookies_path:
|
if cookies_path:
|
||||||
config["extractor"]["cookies"] = cookies_path
|
config["extractor"]["cookies"] = cookies_path
|
||||||
if auth_token and platform == "discord":
|
|
||||||
config["extractor"].setdefault("discord", {})
|
|
||||||
config["extractor"]["discord"]["token"] = auth_token
|
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile(
|
with tempfile.NamedTemporaryFile(
|
||||||
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
|
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
|
||||||
@@ -1004,8 +937,6 @@ class GalleryDLService:
|
|||||||
config = self._build_config_for_source(platform, source_config, artist_slug)
|
config = self._build_config_for_source(platform, source_config, artist_slug)
|
||||||
if cookies_path:
|
if cookies_path:
|
||||||
config["extractor"]["cookies"] = cookies_path
|
config["extractor"]["cookies"] = cookies_path
|
||||||
if auth_token and platform == "discord":
|
|
||||||
config["extractor"].setdefault("discord", {})["token"] = auth_token
|
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile(
|
with tempfile.NamedTemporaryFile(
|
||||||
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
|
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
|
||||||
|
|||||||
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
|
|||||||
return active
|
return active
|
||||||
|
|
||||||
|
|
||||||
|
def backfill_ready(source: Source) -> bool:
|
||||||
|
"""A deep walk the operator started, with budget left and no failure
|
||||||
|
backing it off — due NOW rather than at its next scheduled check.
|
||||||
|
|
||||||
|
A backfill runs one time-boxed chunk per download (plan #693), and nothing
|
||||||
|
queued the next chunk: each waited for the source's regular interval. At
|
||||||
|
the 8-hour default a freshly armed backfill sat untouched until the next
|
||||||
|
check (the operator armed one on 2026-09-25 and saw nothing happen) and a
|
||||||
|
five-chunk walk took most of two days. The tick's in-flight guard keeps
|
||||||
|
one chunk at a time per source and the platform lock one walk per
|
||||||
|
platform, so "due every tick" means "next chunk as soon as the last one
|
||||||
|
ends".
|
||||||
|
|
||||||
|
The failure gate is what keeps a broken source from retrying every
|
||||||
|
minute: any failed chunk raises `consecutive_failures`, which drops the
|
||||||
|
source back onto its backed-off interval. A chunk that fails to progress
|
||||||
|
twice marks the walk stalled (download_service), which ends it here too.
|
||||||
|
"""
|
||||||
|
co = source.config_overrides or {}
|
||||||
|
return (
|
||||||
|
co.get("_backfill_state") == "running"
|
||||||
|
and (source.backfill_runs_remaining or 0) > 0
|
||||||
|
and not (source.consecutive_failures or 0)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def select_due_sources(session: AsyncSession) -> list[Source]:
|
async def select_due_sources(session: AsyncSession) -> list[Source]:
|
||||||
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
|
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
|
||||||
|
|
||||||
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
|||||||
cooldown is the preventive half of the burst-prevention pair (per-source
|
cooldown is the preventive half of the burst-prevention pair (per-source
|
||||||
consecutive_failures backoff handles the offending source itself).
|
consecutive_failures backoff handles the offending source itself).
|
||||||
|
|
||||||
|
A running backfill (`backfill_ready`) is due on every tick, and whether
|
||||||
|
or not its artist is on auto-check — the operator started it by hand.
|
||||||
|
|
||||||
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
|
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
|
||||||
sources go first, then the longest-since-checked, so the most overdue
|
sources go first, then the longest-since-checked, so the most overdue
|
||||||
sources hit Celery's FIFO download queue first. Anti-starvation: if
|
sources hit Celery's FIFO download queue first. Anti-starvation: if
|
||||||
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
|||||||
.options(selectinload(Source.artist))
|
.options(selectinload(Source.artist))
|
||||||
.join(Artist, Source.artist_id == Artist.id)
|
.join(Artist, Source.artist_id == Artist.id)
|
||||||
.where(Source.enabled.is_(True))
|
.where(Source.enabled.is_(True))
|
||||||
.where(Artist.auto_check.is_(True))
|
|
||||||
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
|
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
|
||||||
)).scalars().all()
|
)).scalars().all()
|
||||||
|
|
||||||
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
|||||||
for s in rows:
|
for s in rows:
|
||||||
if s.platform in cooldowns:
|
if s.platform in cooldowns:
|
||||||
continue
|
continue
|
||||||
|
if backfill_ready(s):
|
||||||
|
due.append(s)
|
||||||
|
continue
|
||||||
|
if not s.artist.auto_check:
|
||||||
|
continue
|
||||||
interval = compute_effective_interval(s, s.artist, settings)
|
interval = compute_effective_interval(s, s.artist, settings)
|
||||||
if s.last_checked_at is None:
|
if s.last_checked_at is None:
|
||||||
due.append(s)
|
due.append(s)
|
||||||
@@ -161,6 +194,8 @@ def compute_next_check_at(
|
|||||||
source: Source, artist: Artist, settings: ImportSettings,
|
source: Source, artist: Artist, settings: ImportSettings,
|
||||||
) -> datetime | None:
|
) -> datetime | None:
|
||||||
"""Return the projected datetime of the next check, or None if never checked."""
|
"""Return the projected datetime of the next check, or None if never checked."""
|
||||||
|
if backfill_ready(source):
|
||||||
|
return datetime.now(UTC)
|
||||||
if source.last_checked_at is None:
|
if source.last_checked_at is None:
|
||||||
return None
|
return None
|
||||||
interval = compute_effective_interval(source, artist, settings)
|
interval = compute_effective_interval(source, artist, settings)
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ from ..models import (
|
|||||||
Source,
|
Source,
|
||||||
)
|
)
|
||||||
from .db_helpers import failing_sources_clause
|
from .db_helpers import failing_sources_clause
|
||||||
|
from .download_backends import uses_native_ingester
|
||||||
from .gallery_dl import ErrorType
|
from .gallery_dl import ErrorType
|
||||||
from .membership_reconcile import KEPT_KEY, STOPPED_KEY
|
from .membership_reconcile import KEPT_KEY, STOPPED_KEY
|
||||||
from .membership_roster import gated_reasons_for_sources
|
from .membership_roster import gated_reasons_for_sources
|
||||||
@@ -125,6 +126,11 @@ class SourceRecord:
|
|||||||
"backfill_posts": self.backfill_posts,
|
"backfill_posts": self.backfill_posts,
|
||||||
"tier_gated_count": self.tier_gated_count,
|
"tier_gated_count": self.tier_gated_count,
|
||||||
"gated_reason": self.gated_reason,
|
"gated_reason": self.gated_reason,
|
||||||
|
# Recover / recapture exist only on the native ingester. Sent so the
|
||||||
|
# UI asks the backend's own predicate instead of keeping a copy of
|
||||||
|
# the platform list — the copy said "patreon, subscribestar" for a
|
||||||
|
# day after Discord went native (milestone 428).
|
||||||
|
"native_ingester": uses_native_ingester(self.platform),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -551,8 +557,8 @@ class SourceService:
|
|||||||
whole source); the two flags are mutually exclusive, so arming recapture
|
whole source); the two flags are mutually exclusive, so arming recapture
|
||||||
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
|
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
|
||||||
fresh from the top. The flag is cleared on completion (download_service)
|
fresh from the top. The flag is cleared on completion (download_service)
|
||||||
and on stop. Recapture is Patreon-only (the native ingester's post-record
|
and on stop. Recapture needs the native ingester's post-record capture,
|
||||||
capture); inert elsewhere. The UI gates the action to Patreon sources."""
|
so the UI offers it on native sources only (`native_ingester`)."""
|
||||||
source = (await self.session.execute(
|
source = (await self.session.execute(
|
||||||
select(Source).where(Source.id == source_id)
|
select(Source).where(Source.id == source_id)
|
||||||
)).scalar_one_or_none()
|
)).scalar_one_or_none()
|
||||||
|
|||||||
@@ -76,10 +76,10 @@ const running = computed(() => props.source.backfill_state === 'running')
|
|||||||
const recovering = computed(() => !!props.source.backfill_bypass_seen)
|
const recovering = computed(() => !!props.source.backfill_bypass_seen)
|
||||||
const recapturing = computed(() => !!props.source.backfill_recapture)
|
const recapturing = computed(() => !!props.source.backfill_recapture)
|
||||||
// Recover / recapture are native-ingester features (ledger-bypass re-walk and
|
// Recover / recapture are native-ingester features (ledger-bypass re-walk and
|
||||||
// post-text re-grab), available to every native platform — not just Patreon.
|
// post-text re-grab), available on every native platform. The backend says
|
||||||
// Mirrors backend download_backends.NATIVE_INGESTER_PLATFORMS.
|
// which those are (`native_ingester`); a copied list here went stale when
|
||||||
const NATIVE_PLATFORMS = ['patreon', 'subscribestar']
|
// Discord moved over.
|
||||||
const isNative = computed(() => NATIVE_PLATFORMS.includes(props.source.platform))
|
const isNative = computed(() => !!props.source.native_ingester)
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
|
|||||||
@@ -679,7 +679,7 @@ async function onRecover(source) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
|
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
|
||||||
// external links and localize on-disk inline images, without re-downloading
|
// external links and localize on-disk inline images, without re-downloading
|
||||||
// media. Reuses the backfill lifecycle/badge; stop via the same Stop control.
|
// media. Reuses the backfill lifecycle/badge; stop via the same Stop control.
|
||||||
async function onRecapture(source) {
|
async function onRecapture(source) {
|
||||||
|
|||||||
@@ -140,7 +140,7 @@ export const useSourcesStore = defineStore('sources', () => {
|
|||||||
_patchSource(body)
|
_patchSource(body)
|
||||||
return body
|
return body
|
||||||
}
|
}
|
||||||
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
|
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
|
||||||
// external links and localize on-disk inline images, WITHOUT re-downloading
|
// external links and localize on-disk inline images, WITHOUT re-downloading
|
||||||
// media. Shares the backfill lifecycle/badge; stop via stopBackfill.
|
// media. Shares the backfill lifecycle/badge; stop via stopBackfill.
|
||||||
async function recaptureSource(id, artistIdHint = null) {
|
async function recaptureSource(id, artistIdHint = null) {
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""Discord file naming, checked against the metadata gallery-dl really emitted.
|
||||||
|
|
||||||
|
Discord moved from gallery-dl to the native downloader in milestone 428. Every
|
||||||
|
file gallery-dl wrote is on disk under gallery-dl's naming, and the native
|
||||||
|
downloader finds it there — skips it instead of fetching it again — only if it
|
||||||
|
builds the SAME name. This file used to pin gallery-dl's patterns (#3999: they
|
||||||
|
named keys the extractor never set, and every file landed as
|
||||||
|
`None/<date>_None_<name>`); it now pins the native downloader to what those
|
||||||
|
patterns produced from the same real metadata.
|
||||||
|
|
||||||
|
The fixture keeps the key set and value types of a real attachment sidecar
|
||||||
|
from the operator's instance (2026-09-13), with every value invented.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from backend.app.services.discord_client import DiscordClient
|
||||||
|
from backend.app.services.discord_downloader import channel_dir, media_stem
|
||||||
|
from backend.app.utils.sidecar import find_sidecar
|
||||||
|
|
||||||
|
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _message_from_the_fixture():
|
||||||
|
"""The API message that produced the fixture's sidecar."""
|
||||||
|
gdl = json.loads(_FIXTURE.read_text())
|
||||||
|
date, time = gdl["date"].split(" ")
|
||||||
|
return {
|
||||||
|
"id": gdl["message_id"],
|
||||||
|
"type": 0,
|
||||||
|
"timestamp": f"{date}T{time}.000000+00:00",
|
||||||
|
"content": gdl["message"],
|
||||||
|
"attachments": [{"id": "500000000000000005", "url": gdl["url"]}],
|
||||||
|
"embeds": [],
|
||||||
|
"_meta": {"channel": gdl["channel"], "channel_id": gdl["channel_id"],
|
||||||
|
"server": gdl["server"], "server_id": gdl["server_id"]},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_native_name_is_the_name_gallery_dl_wrote(tmp_path):
|
||||||
|
"""gallery-dl's `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`
|
||||||
|
in directory `{channel}`, rendered from the fixture."""
|
||||||
|
msg = _message_from_the_fixture()
|
||||||
|
[media] = DiscordClient.extract_media(msg)
|
||||||
|
assert media_stem(msg, media) + "." + media.extension == (
|
||||||
|
"20240716_300000000000000003_01_image.png"
|
||||||
|
)
|
||||||
|
assert channel_dir(tmp_path, "a", msg) == tmp_path / "a" / "discord" / "nsfw-drops"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_gallery_dl_sidecar_on_disk_still_pairs_with_its_file(tmp_path):
|
||||||
|
"""Files gallery-dl wrote keep their full sidecars beside them; the native
|
||||||
|
name must be the stem those sidecars were named after."""
|
||||||
|
msg = _message_from_the_fixture()
|
||||||
|
[media] = DiscordClient.extract_media(msg)
|
||||||
|
stem = media_stem(msg, media)
|
||||||
|
(tmp_path / f"{stem}.png").write_bytes(b"x")
|
||||||
|
(tmp_path / f"{stem}.json").write_text(_FIXTURE.read_text())
|
||||||
|
assert find_sidecar(tmp_path / f"{stem}.png") == tmp_path / f"{stem}.json"
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_files_with_the_same_original_name_get_distinct_names():
|
||||||
|
"""`image.png` is Discord's default name; two messages must not collide."""
|
||||||
|
one = _message_from_the_fixture()
|
||||||
|
two = {**one, "id": "300000000000000099"}
|
||||||
|
[m1] = DiscordClient.extract_media(one)
|
||||||
|
[m2] = DiscordClient.extract_media(two)
|
||||||
|
assert media_stem(one, m1) != media_stem(two, m2)
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
"""gallery-dl naming patterns, rendered against the metadata gallery-dl really emits.
|
|
||||||
|
|
||||||
gallery-dl does not fail on a format field that names a key the extractor never
|
|
||||||
sets. It renders "None" and carries on. That is how every Discord download
|
|
||||||
landed in a `None/` directory as `<date>_None_<name>`: the patterns asked for
|
|
||||||
`{channel[name]}` and `{id}`, and the real metadata has a string `channel` and
|
|
||||||
`message_id`, with no `id` at all. The mismatch also broke sidecar pairing, so
|
|
||||||
no Discord file ever got a Post or a post date.
|
|
||||||
|
|
||||||
The fixture keeps the key set and value TYPES of a real attachment sidecar from
|
|
||||||
the operator's instance (2026-09-13), with every value invented. Rendering
|
|
||||||
through Python's own formatter raises on a missing key or a subscript into a
|
|
||||||
string, which is the loud failure gallery-dl does not give.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import string
|
|
||||||
from datetime import datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from backend.app.services.gallery_dl import (
|
|
||||||
DISCORD_DIRECTORY,
|
|
||||||
DISCORD_FILENAME,
|
|
||||||
GalleryDLService,
|
|
||||||
SourceConfig,
|
|
||||||
sidecar_name_for,
|
|
||||||
)
|
|
||||||
from backend.app.utils.sidecar import find_sidecar
|
|
||||||
|
|
||||||
_FIXTURE = Path(__file__).parent / "fixtures" / "discord_attachment_sidecar.json"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def kwdict():
|
|
||||||
data = json.loads(_FIXTURE.read_text())
|
|
||||||
# gallery-dl hands the formatter a datetime; the JSON sidecar stores it as text.
|
|
||||||
data["date"] = datetime.strptime(data["date"], "%Y-%m-%d %H:%M:%S")
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def _render(pattern, kwdict):
|
|
||||||
return string.Formatter().vformat(pattern, (), kwdict)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_discord_filename_renders_from_real_keys(kwdict):
|
|
||||||
name = _render(DISCORD_FILENAME, kwdict)
|
|
||||||
assert "None" not in name
|
|
||||||
assert name == "20240716_300000000000000003_01_image.png"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_discord_directory_renders_from_real_keys(kwdict):
|
|
||||||
assert [_render(p, kwdict) for p in DISCORD_DIRECTORY] == ["nsfw-drops"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"broken", ["{channel[name]}", "{date:%Y%m%d}_{id}_{filename}.{extension}"],
|
|
||||||
)
|
|
||||||
def test_the_patterns_that_shipped_would_have_failed_here(kwdict, broken):
|
|
||||||
"""Positive control: the two patterns that produced `None/..._None_...`
|
|
||||||
must fail this renderer, or the tests above prove nothing."""
|
|
||||||
with pytest.raises((KeyError, TypeError)):
|
|
||||||
_render(broken, kwdict)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_sidecar_is_named_exactly_like_its_media(kwdict, tmp_path):
|
|
||||||
"""So `find_sidecar` pairs them on its first candidate, and two attachments
|
|
||||||
that share an original name (`image.png`) can never share a sidecar."""
|
|
||||||
media = tmp_path / _render(DISCORD_FILENAME, kwdict)
|
|
||||||
sidecar = tmp_path / _render(sidecar_name_for(DISCORD_FILENAME), kwdict)
|
|
||||||
media.write_bytes(b"x")
|
|
||||||
sidecar.write_text("{}")
|
|
||||||
assert find_sidecar(media) == sidecar
|
|
||||||
|
|
||||||
|
|
||||||
def test_two_attachments_with_the_same_original_name_get_distinct_sidecars(kwdict):
|
|
||||||
other = {**kwdict, "message_id": "300000000000000099"}
|
|
||||||
pattern = sidecar_name_for(DISCORD_FILENAME)
|
|
||||||
assert _render(pattern, kwdict) != _render(pattern, other)
|
|
||||||
|
|
||||||
|
|
||||||
def test_sidecar_name_for_needs_an_extension_suffix():
|
|
||||||
assert sidecar_name_for("{a}_{b}.{extension}") == "{a}_{b}.json"
|
|
||||||
assert sidecar_name_for("{a}_{b}") is None
|
|
||||||
|
|
||||||
|
|
||||||
# --- the config gallery-dl is actually given -----------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def gdl(tmp_path):
|
|
||||||
return GalleryDLService(images_root=tmp_path / "images", validate_files=False)
|
|
||||||
|
|
||||||
|
|
||||||
def _discord_section(gdl, **overrides):
|
|
||||||
cfg = gdl._build_config_for_source(
|
|
||||||
platform="discord", source_config=SourceConfig(**overrides), artist_slug="a",
|
|
||||||
)
|
|
||||||
return cfg["extractor"]["discord"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_discord_config_carries_the_mirrored_sidecar(gdl):
|
|
||||||
section = _discord_section(gdl)
|
|
||||||
assert section["filename"] == DISCORD_FILENAME
|
|
||||||
assert section["directory"] == DISCORD_DIRECTORY
|
|
||||||
assert section["postprocessors"][0]["filename"] == sidecar_name_for(DISCORD_FILENAME)
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_filename_override_keeps_the_sidecar_mirrored(gdl):
|
|
||||||
section = _discord_section(gdl, filename_pattern="{message_id}_{num}.{extension}")
|
|
||||||
assert section["postprocessors"][0]["filename"] == "{message_id}_{num}.json"
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_metadata_means_no_discord_postprocessor(gdl):
|
|
||||||
section = _discord_section(gdl, save_metadata=False)
|
|
||||||
assert "postprocessors" not in section
|
|
||||||
@@ -11,6 +11,7 @@ import pytest
|
|||||||
|
|
||||||
from backend.app.models import Artist, ImportSettings, Source
|
from backend.app.models import Artist, ImportSettings, Source
|
||||||
from backend.app.services.scheduler_service import (
|
from backend.app.services.scheduler_service import (
|
||||||
|
backfill_ready,
|
||||||
compute_effective_interval,
|
compute_effective_interval,
|
||||||
scheduler_status,
|
scheduler_status,
|
||||||
select_due_sources,
|
select_due_sources,
|
||||||
@@ -150,6 +151,67 @@ async def test_select_includes_past_due(db):
|
|||||||
assert any(s.url == "https://sel-past" for s in due)
|
assert any(s.url == "https://sel-past" for s in due)
|
||||||
|
|
||||||
|
|
||||||
|
# --- a running backfill is due every tick (2026-09-25) ---------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _backfilling(runs=5, failures=0, state="running"):
|
||||||
|
src = _src(failures=failures)
|
||||||
|
src.config_overrides = {"_backfill_state": state}
|
||||||
|
src.backfill_runs_remaining = runs
|
||||||
|
return src
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_running_backfill_is_ready():
|
||||||
|
assert backfill_ready(_backfilling()) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_backfill_out_of_budget_or_not_running_is_not_ready():
|
||||||
|
assert backfill_ready(_backfilling(runs=0)) is False
|
||||||
|
assert backfill_ready(_backfilling(state="stalled")) is False
|
||||||
|
assert backfill_ready(_src()) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failing_backfill_falls_back_to_its_backoff():
|
||||||
|
"""Otherwise a source whose chunks keep erroring would retry every minute."""
|
||||||
|
assert backfill_ready(_backfilling(failures=1)) is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_select_runs_a_backfill_now_not_at_its_next_check(db):
|
||||||
|
"""Checked a minute ago on an hourly interval: not due, unless backfilling —
|
||||||
|
the next chunk should not wait an interval (the 2026-09-25 armed-and-idle
|
||||||
|
backfill)."""
|
||||||
|
artist = await _seed_artist(db, interval=3600, name="bf-now")
|
||||||
|
for url, overrides, runs in (
|
||||||
|
("https://bf-running", {"_backfill_state": "running"}, 5),
|
||||||
|
("https://bf-idle", {}, 0),
|
||||||
|
):
|
||||||
|
db.add(Source(
|
||||||
|
artist_id=artist.id, platform="patreon", url=url, enabled=True,
|
||||||
|
consecutive_failures=0, config_overrides=overrides,
|
||||||
|
backfill_runs_remaining=runs,
|
||||||
|
last_checked_at=datetime.now(UTC) - timedelta(seconds=60),
|
||||||
|
))
|
||||||
|
await db.commit()
|
||||||
|
urls = {s.url for s in await select_due_sources(db)}
|
||||||
|
assert "https://bf-running" in urls
|
||||||
|
assert "https://bf-idle" not in urls
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_backfill_runs_even_when_its_artist_is_not_auto_checked(db):
|
||||||
|
"""The operator started it by hand; auto-check governs the schedule only."""
|
||||||
|
artist = await _seed_artist(db, auto=False, name="bf-noauto")
|
||||||
|
db.add(Source(
|
||||||
|
artist_id=artist.id, platform="patreon", url="https://bf-noauto",
|
||||||
|
enabled=True, consecutive_failures=0,
|
||||||
|
config_overrides={"_backfill_state": "running"}, backfill_runs_remaining=5,
|
||||||
|
last_checked_at=datetime.now(UTC),
|
||||||
|
))
|
||||||
|
await db.commit()
|
||||||
|
assert any(s.url == "https://bf-noauto" for s in await select_due_sources(db))
|
||||||
|
|
||||||
|
|
||||||
# --- platform-rate-limit cooldown -----------------------------------------
|
# --- platform-rate-limit cooldown -----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -34,6 +34,24 @@ async def test_known_platforms_are_the_supported_four(db):
|
|||||||
assert "pixiv" not in KNOWN_PLATFORMS
|
assert "pixiv" not in KNOWN_PLATFORMS
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_source_says_whether_it_is_on_the_native_ingester(db):
|
||||||
|
"""The UI offers recover / recapture from this flag rather than a copied
|
||||||
|
platform list — the copy went stale when Discord moved (milestone 428)."""
|
||||||
|
artist = await _artist(db, name="Natty")
|
||||||
|
svc = SourceService(db)
|
||||||
|
discord = await svc.create(
|
||||||
|
artist_id=artist.id, platform="discord",
|
||||||
|
url="https://discord.com/channels/1/2",
|
||||||
|
)
|
||||||
|
hf = await svc.create(
|
||||||
|
artist_id=artist.id, platform="hentaifoundry",
|
||||||
|
url="https://www.hentai-foundry.com/user/Natty",
|
||||||
|
)
|
||||||
|
assert discord.to_dict()["native_ingester"] is True
|
||||||
|
assert hf.to_dict()["native_ingester"] is False
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_create_flips_is_subscription_on_first_source(db):
|
async def test_create_flips_is_subscription_on_first_source(db):
|
||||||
artist = await _artist(db)
|
artist = await _artist(db)
|
||||||
|
|||||||
Reference in New Issue
Block a user