feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)

Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 15:02:46 -04:00
parent add1c1ad14
commit 96fffaff64
10 changed files with 365 additions and 310 deletions
@@ -26,8 +26,8 @@ from sqlalchemy.orm import joinedload
 from ..models import Artist, DownloadEvent, Source
 from .credential_service import CredentialService
 from .gallery_dl import (
+    BACKFILL_CHUNK_SECONDS,
    BACKFILL_SKIP_VALUE,
-    BACKFILL_TIMEOUT_SECONDS,
    TICK_SKIP_VALUE,
    ErrorType,
    GalleryDLService,
@@ -108,23 +108,20 @@ class DownloadService:
        self.sync_session.close()

        source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
-        # alembic 0031 / plan #544: derive skip_value + timeout from the
-        # source's backfill_runs_remaining counter. When > 0, walk the full
-        # post history (skip: True + 1170s); when 0, exit gallery-dl after
-        # 20 contiguous archived items (skip: "exit:20" + the default
-        # 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
+        # Backfill mode (plan #693): the source's `_backfill_state == "running"`
+        # selects a time-boxed deep-walk chunk — skip: True (walk full history)
+        # + the BACKFILL_CHUNK_SECONDS budget, resuming from the cursor
+        # checkpoint (plan #689). State is the single source of truth; it stays
+        # "running" across ticks until the walk reaches the bottom (phase 3
+        # flips it to "complete"). Otherwise tick mode: exit gallery-dl after
+        # 20 contiguous archived items (skip: "exit:20" + the default 870s).
+        # Operator drives this via POST /api/sources/{id}/backfill {action}.
        overrides = ctx["config_overrides"] or {}
-        backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
-        # Cursor-paged backfill (plan #689): a pending checkpoint keeps the
-        # deep-walk in backfill mode across ticks until it reaches the
-        # bottom, even after the operator's run budget hits 0 — so one large
-        # creator (Anduo) finishes without re-triggering. The phase-3
-        # stuck-guard bounds it. patreon-only: it's the sole platform with a
-        # resumable gallery-dl cursor.
-        pending_cursor = overrides.get("_backfill_cursor")
-        if backfill_remaining > 0 or pending_cursor:
+        in_backfill = overrides.get("_backfill_state") == "running"
+        if in_backfill:
            skip_value: bool | str = BACKFILL_SKIP_VALUE
-            source_config.timeout = BACKFILL_TIMEOUT_SECONDS
+            source_config.timeout = BACKFILL_CHUNK_SECONDS
+            pending_cursor = overrides.get("_backfill_cursor")
            if ctx["platform"] == "patreon" and pending_cursor:
                source_config.resume_cursor = pending_cursor
        else:
@@ -171,6 +168,24 @@ class DownloadService:
                        skip_value=skip_value,
                    )

+        # A backfill chunk that hit its time-box but made forward progress is
+        # NORMAL, not a failure — reclassify TIMEOUT → PARTIAL so it reads as
+        # "ok/progress" (PARTIAL maps to status "ok"), not a red error, and the
+        # next chunk just resumes from the new cursor (plan #693). A chunk that
+        # timed out with NO progress stays TIMEOUT and feeds phase 3's
+        # stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires.
+        if in_backfill and dl_result.error_type == ErrorType.TIMEOUT:
+            new_cursor = parse_last_cursor(dl_result.stdout, dl_result.stderr)
+            advanced = bool(
+                (new_cursor and new_cursor != overrides.get("_backfill_cursor"))
+                or dl_result.files_downloaded > 0
+            )
+            if advanced:
+                dl_result.error_type = ErrorType.PARTIAL
+                dl_result.error_message = (
+                    f"Backfill chunk: {dl_result.files_downloaded} file(s) — continuing"
+                )
+
        return await self._phase3_persist(
            ctx["event_id"], ctx, dl_result, resolved_campaign_id,
        )
@@ -411,35 +426,38 @@ class DownloadService:
        return event_id

    async def _apply_backfill_lifecycle(self, ctx: dict, dl_result) -> None:
-        """Cursor-paged backfill state machine (plan #689).
+        """Backfill state machine (plan #693, building on the cursor of #689).

-        A backfill is "in progress" while backfill_runs_remaining > 0 OR a
-        checkpoint cursor is stored. Completion is a CLEAN rc=0 finish —
-        gallery-dl with skip:True exits 0 only after exhausting the
-        newest→oldest walk; a run cut short by the subprocess budget returns
-        success=False / return_code=-1 (the TimeoutExpired path), so rc=0 is
-        an unambiguous "reached the bottom" signal regardless of how many
-        files this run fetched. (This replaces plan #544's files==0 predicate,
-        whose audit-2026-06-02 VALIDATION_FAILED guard still holds: those exit
-        non-zero or carry an error_type, so they are not "completed".)
+        A backfill runs in time-boxed chunks while
+        `config_overrides["_backfill_state"] == "running"`. Each chunk:
+          - COMPLETES the walk → clean rc=0 (gallery-dl with skip:True exits 0
+            only after exhausting the newest→oldest walk; a chunk cut short by
+            its time-box returns success=False / rc<0 via TimeoutExpired). On
+            completion: state="complete", clear the cursor, return to tick mode.
+          - made PROGRESS (cursor advanced and/or files written) → stay
+            "running", checkpoint the new cursor, bump the chunk counter, and
+            spend one of the safety-cap chunks (backfill_runs_remaining). If the
+            cap is exhausted without finishing → state="stalled".
+          - made NO progress → increment the stall counter; two strikes →
+            state="stalled", clear the cursor (a wedged walk can't loop).

-        For patreon, each non-completing run checkpoints gallery-dl's
-        last-emitted pagination cursor — even a timed-out run, since its
-        partial stdout/stderr still carries it — so the next run RESUMES
-        instead of re-walking from the top. A stuck-guard clears the cursor
-        after two consecutive runs that fail to advance it, so a wedged walk
-        can't re-strand forever (cf. download soft-limit ladder).
+        State is the single source of truth; the cursor lives only inside a
+        running backfill. config_overrides is reassigned (not mutated in place)
+        for SQLAlchemy JSON change detection.
        """
        overrides = ctx["config_overrides"] or {}
+        if overrides.get("_backfill_state") != "running":
+            return  # not backfilling — tick mode, nothing to do
+
        old_cursor = overrides.get("_backfill_cursor")
-        backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
-        if backfill_remaining <= 0 and not old_cursor:
-            return  # tick mode — counter untouched
+        cap_remaining = ctx.get("backfill_runs_remaining", 0) or 0

        src = (await self.async_session.execute(
            select(Source).where(Source.id == ctx["source_id"])
        )).scalar_one()
        new_overrides = dict(src.config_overrides or {})
+        chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
+        new_overrides["_backfill_chunks"] = chunks

        completed = (
            dl_result.success
@@ -447,33 +465,43 @@ class DownloadService:
            and dl_result.return_code == 0
        )
        if completed:
+            new_overrides["_backfill_state"] = "complete"
            new_overrides.pop("_backfill_cursor", None)
            new_overrides.pop("_backfill_cursor_stalls", None)
            src.config_overrides = new_overrides
            src.backfill_runs_remaining = 0
            return

-        # Non-completing run. Patreon checkpoints + resumes via cursor;
-        # other platforms have no resumable cursor, so just decrement.
-        if ctx["platform"] == "patreon":
-            new_cursor = parse_last_cursor(dl_result.stdout, dl_result.stderr)
-            if new_cursor and new_cursor != old_cursor:
+        # Did not finish. Patreon checkpoints + resumes via cursor; other
+        # platforms have no resumable cursor (every chunk re-walks from the
+        # top), so they advance only by the download archive growing.
+        new_cursor = (
+            parse_last_cursor(dl_result.stdout, dl_result.stderr)
+            if ctx["platform"] == "patreon" else None
+        )
+        advanced = bool(
+            (new_cursor and new_cursor != old_cursor)
+            or dl_result.files_downloaded > 0
+        )
+        if advanced:
+            if new_cursor:
                new_overrides["_backfill_cursor"] = new_cursor
+            new_overrides.pop("_backfill_cursor_stalls", None)
+            cap_remaining = max(0, cap_remaining - 1)
+            src.backfill_runs_remaining = cap_remaining
+            if cap_remaining == 0:
+                # Safety cap hit before reaching the bottom — pause, don't loop.
+                new_overrides["_backfill_state"] = "stalled"
+        else:
+            stalls = int(new_overrides.get("_backfill_cursor_stalls", 0)) + 1
+            if stalls >= 2:
+                new_overrides["_backfill_state"] = "stalled"
+                new_overrides.pop("_backfill_cursor", None)
                new_overrides.pop("_backfill_cursor_stalls", None)
            else:
-                stalls = int(new_overrides.get("_backfill_cursor_stalls", 0)) + 1
-                if stalls >= 2:
-                    # Wedged: give up, drop back to tick mode. The surfaced
-                    # error_type on the event tells the operator why.
-                    new_overrides.pop("_backfill_cursor", None)
-                    new_overrides.pop("_backfill_cursor_stalls", None)
-                    src.config_overrides = new_overrides
-                    src.backfill_runs_remaining = 0
-                    return
                new_overrides["_backfill_cursor_stalls"] = stalls
-            src.config_overrides = new_overrides

-        src.backfill_runs_remaining = max(0, backfill_remaining - 1)
+        src.config_overrides = new_overrides

    async def _update_source_health(
        self, *, source_id: int, status: str, error_message: str | None,