feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

- Recover and Recapture show on every native source. The menu gated them on
  a copied platform list ('patreon', 'subscribestar') that went stale when
  Discord moved over. Sources now carry `native_ingester` from the backend's
  own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
  backfill's next chunk: each one waited for the source's regular interval,
  so an armed backfill sat idle until the next check (8h at the default) and
  a five-chunk walk took most of two days. The in-flight guard and the
  platform lock keep one chunk at a time. A failing source falls back to its
  backoff, and a stalled or out-of-budget walk stops being due. It also runs
  when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
  defaults, sidecar-mirroring postprocessor and token injection are gone.
  The naming test moves to the native downloader and still renders against
  the real gallery-dl sidecar fixture. That is the guard that the files
  gallery-dl wrote are found on disk rather than fetched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 23:14:48 -04:00
co-authored by Claude Opus 5.5
parent 84e5448941
commit e5bdcd2596
11 changed files with 208 additions and 202 deletions
+62
View File
@@ -11,6 +11,7 @@ import pytest
from backend.app.models import Artist, ImportSettings, Source
from backend.app.services.scheduler_service import (
backfill_ready,
compute_effective_interval,
scheduler_status,
select_due_sources,
@@ -150,6 +151,67 @@ async def test_select_includes_past_due(db):
assert any(s.url == "https://sel-past" for s in due)
# --- a running backfill is due every tick (2026-09-25) ---------------------
def _backfilling(runs=5, failures=0, state="running"):
src = _src(failures=failures)
src.config_overrides = {"_backfill_state": state}
src.backfill_runs_remaining = runs
return src
def test_a_running_backfill_is_ready():
assert backfill_ready(_backfilling()) is True
def test_a_backfill_out_of_budget_or_not_running_is_not_ready():
assert backfill_ready(_backfilling(runs=0)) is False
assert backfill_ready(_backfilling(state="stalled")) is False
assert backfill_ready(_src()) is False
def test_a_failing_backfill_falls_back_to_its_backoff():
"""Otherwise a source whose chunks keep erroring would retry every minute."""
assert backfill_ready(_backfilling(failures=1)) is False
@pytest.mark.asyncio
async def test_select_runs_a_backfill_now_not_at_its_next_check(db):
"""Checked a minute ago on an hourly interval: not due, unless backfilling —
the next chunk should not wait an interval (the 2026-09-25 armed-and-idle
backfill)."""
artist = await _seed_artist(db, interval=3600, name="bf-now")
for url, overrides, runs in (
("https://bf-running", {"_backfill_state": "running"}, 5),
("https://bf-idle", {}, 0),
):
db.add(Source(
artist_id=artist.id, platform="patreon", url=url, enabled=True,
consecutive_failures=0, config_overrides=overrides,
backfill_runs_remaining=runs,
last_checked_at=datetime.now(UTC) - timedelta(seconds=60),
))
await db.commit()
urls = {s.url for s in await select_due_sources(db)}
assert "https://bf-running" in urls
assert "https://bf-idle" not in urls
@pytest.mark.asyncio
async def test_a_backfill_runs_even_when_its_artist_is_not_auto_checked(db):
"""The operator started it by hand; auto-check governs the schedule only."""
artist = await _seed_artist(db, auto=False, name="bf-noauto")
db.add(Source(
artist_id=artist.id, platform="patreon", url="https://bf-noauto",
enabled=True, consecutive_failures=0,
config_overrides={"_backfill_state": "running"}, backfill_runs_remaining=5,
last_checked_at=datetime.now(UTC),
))
await db.commit()
assert any(s.url == "https://bf-noauto" for s in await select_due_sources(db))
# --- platform-rate-limit cooldown -----------------------------------------