feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -16,7 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from ..models import Artist, Source
|
||||
from ..utils.slug import slugify
|
||||
from .source_service import NEW_SOURCE_BACKFILL_RUNS
|
||||
from .source_service import BACKFILL_MAX_CHUNKS
|
||||
|
||||
|
||||
class UnknownPlatformError(Exception):
|
||||
@@ -205,16 +205,17 @@ class ExtensionService:
|
||||
return existing, False
|
||||
sp = await self.session.begin_nested()
|
||||
try:
|
||||
# New subscription sources arm a few backfill runs so the
|
||||
# first ticks walk the full history (otherwise gallery-dl's
|
||||
# exit:20 short-circuits before the archive is built).
|
||||
# Mirrors SourceService.create — without it, Firefox quick-
|
||||
# add on a creator with >20 unsynced posts would surface
|
||||
# as "check failed" with no diagnosis. Audit 2026-06-02.
|
||||
# New subscription sources arm run-until-done backfill (plan #693)
|
||||
# so the first ticks walk the full history (otherwise gallery-dl's
|
||||
# exit:20 short-circuits before the archive is built). Mirrors
|
||||
# SourceService.create — without it, Firefox quick-add on a creator
|
||||
# with >20 unsynced posts would surface as "check failed" with no
|
||||
# diagnosis. Audit 2026-06-02.
|
||||
src = Source(
|
||||
artist_id=artist_id, platform=platform,
|
||||
url=url, enabled=True,
|
||||
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
|
||||
config_overrides={"_backfill_state": "running"},
|
||||
backfill_runs_remaining=BACKFILL_MAX_CHUNKS,
|
||||
)
|
||||
self.session.add(src)
|
||||
await self.session.flush()
|
||||
|
||||
Reference in New Issue
Block a user