feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

- Recover and Recapture show on every native source. The menu gated them on
  a copied platform list ('patreon', 'subscribestar') that went stale when
  Discord moved over. Sources now carry `native_ingester` from the backend's
  own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
  backfill's next chunk: each one waited for the source's regular interval,
  so an armed backfill sat idle until the next check (8h at the default) and
  a five-chunk walk took most of two days. The in-flight guard and the
  platform lock keep one chunk at a time. A failing source falls back to its
  backoff, and a stalled or out-of-budget walk stops being due. It also runs
  when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
  defaults, sidecar-mirroring postprocessor and token injection are gone.
  The naming test moves to the native downloader and still renders against
  the real gallery-dl sidecar fixture. That is the guard that the files
  gallery-dl wrote are found on disk rather than fetched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 23:14:48 -04:00
co-authored by Claude Opus 5.5
parent 84e5448941
commit e5bdcd2596
11 changed files with 208 additions and 202 deletions
+36 -1
View File
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
return active
def backfill_ready(source: Source) -> bool:
"""A deep walk the operator started, with budget left and no failure
backing it off — due NOW rather than at its next scheduled check.
A backfill runs one time-boxed chunk per download (plan #693), and nothing
queued the next chunk: each waited for the source's regular interval. At
the 8-hour default a freshly armed backfill sat untouched until the next
check (the operator armed one on 2026-09-25 and saw nothing happen) and a
five-chunk walk took most of two days. The tick's in-flight guard keeps
one chunk at a time per source and the platform lock one walk per
platform, so "due every tick" means "next chunk as soon as the last one
ends".
The failure gate is what keeps a broken source from retrying every
minute: any failed chunk raises `consecutive_failures`, which drops the
source back onto its backed-off interval. A chunk that fails to progress
twice marks the walk stalled (download_service), which ends it here too.
"""
co = source.config_overrides or {}
return (
co.get("_backfill_state") == "running"
and (source.backfill_runs_remaining or 0) > 0
and not (source.consecutive_failures or 0)
)
async def select_due_sources(session: AsyncSession) -> list[Source]:
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
cooldown is the preventive half of the burst-prevention pair (per-source
consecutive_failures backoff handles the offending source itself).
A running backfill (`backfill_ready`) is due on every tick, and whether
or not its artist is on auto-check — the operator started it by hand.
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
sources go first, then the longest-since-checked, so the most overdue
sources hit Celery's FIFO download queue first. Anti-starvation: if
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
.options(selectinload(Source.artist))
.join(Artist, Source.artist_id == Artist.id)
.where(Source.enabled.is_(True))
.where(Artist.auto_check.is_(True))
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
)).scalars().all()
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
for s in rows:
if s.platform in cooldowns:
continue
if backfill_ready(s):
due.append(s)
continue
if not s.artist.auto_check:
continue
interval = compute_effective_interval(s, s.artist, settings)
if s.last_checked_at is None:
due.append(s)
@@ -161,6 +194,8 @@ def compute_next_check_at(
source: Source, artist: Artist, settings: ImportSettings,
) -> datetime | None:
"""Return the projected datetime of the next check, or None if never checked."""
if backfill_ready(source):
return datetime.now(UTC)
if source.last_checked_at is None:
return None
interval = compute_effective_interval(source, artist, settings)