feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
a copied platform list ('patreon', 'subscribestar') that went stale when
Discord moved over. Sources now carry `native_ingester` from the backend's
own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
backfill's next chunk: each one waited for the source's regular interval,
so an armed backfill sat idle until the next check (8h at the default) and
a five-chunk walk took most of two days. The in-flight guard and the
platform lock keep one chunk at a time. A failing source falls back to its
backoff, and a stalled or out-of-budget walk stops being due. It also runs
when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
defaults, sidecar-mirroring postprocessor and token injection are gone.
The naming test moves to the native downloader and still renders against
the real gallery-dl sidecar fixture. That is the guard that the files
gallery-dl wrote are found on disk rather than fetched again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
|
||||
return active
|
||||
|
||||
|
||||
def backfill_ready(source: Source) -> bool:
|
||||
"""A deep walk the operator started, with budget left and no failure
|
||||
backing it off — due NOW rather than at its next scheduled check.
|
||||
|
||||
A backfill runs one time-boxed chunk per download (plan #693), and nothing
|
||||
queued the next chunk: each waited for the source's regular interval. At
|
||||
the 8-hour default a freshly armed backfill sat untouched until the next
|
||||
check (the operator armed one on 2026-09-25 and saw nothing happen) and a
|
||||
five-chunk walk took most of two days. The tick's in-flight guard keeps
|
||||
one chunk at a time per source and the platform lock one walk per
|
||||
platform, so "due every tick" means "next chunk as soon as the last one
|
||||
ends".
|
||||
|
||||
The failure gate is what keeps a broken source from retrying every
|
||||
minute: any failed chunk raises `consecutive_failures`, which drops the
|
||||
source back onto its backed-off interval. A chunk that fails to progress
|
||||
twice marks the walk stalled (download_service), which ends it here too.
|
||||
"""
|
||||
co = source.config_overrides or {}
|
||||
return (
|
||||
co.get("_backfill_state") == "running"
|
||||
and (source.backfill_runs_remaining or 0) > 0
|
||||
and not (source.consecutive_failures or 0)
|
||||
)
|
||||
|
||||
|
||||
async def select_due_sources(session: AsyncSession) -> list[Source]:
|
||||
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
|
||||
|
||||
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
||||
cooldown is the preventive half of the burst-prevention pair (per-source
|
||||
consecutive_failures backoff handles the offending source itself).
|
||||
|
||||
A running backfill (`backfill_ready`) is due on every tick, and whether
|
||||
or not its artist is on auto-check — the operator started it by hand.
|
||||
|
||||
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
|
||||
sources go first, then the longest-since-checked, so the most overdue
|
||||
sources hit Celery's FIFO download queue first. Anti-starvation: if
|
||||
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
||||
.options(selectinload(Source.artist))
|
||||
.join(Artist, Source.artist_id == Artist.id)
|
||||
.where(Source.enabled.is_(True))
|
||||
.where(Artist.auto_check.is_(True))
|
||||
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
|
||||
)).scalars().all()
|
||||
|
||||
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
|
||||
for s in rows:
|
||||
if s.platform in cooldowns:
|
||||
continue
|
||||
if backfill_ready(s):
|
||||
due.append(s)
|
||||
continue
|
||||
if not s.artist.auto_check:
|
||||
continue
|
||||
interval = compute_effective_interval(s, s.artist, settings)
|
||||
if s.last_checked_at is None:
|
||||
due.append(s)
|
||||
@@ -161,6 +194,8 @@ def compute_next_check_at(
|
||||
source: Source, artist: Artist, settings: ImportSettings,
|
||||
) -> datetime | None:
|
||||
"""Return the projected datetime of the next check, or None if never checked."""
|
||||
if backfill_ready(source):
|
||||
return datetime.now(UTC)
|
||||
if source.last_checked_at is None:
|
||||
return None
|
||||
interval = compute_effective_interval(source, artist, settings)
|
||||
|
||||
Reference in New Issue
Block a user