feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)

Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).

- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
  far under the 1350 soft limit. Hitting it = normal chunk boundary (the
  TimeoutExpired path already captures partial output + the cursor), not a
  near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
  (running/complete/stalled). A running backfill auto-continues in chunks
  across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
  'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
  pause a pathological walk as 'stalled'. Replaces the N-runs counter
  (backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
  and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
  one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
  start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
  source dict exposes backfill_state + backfill_chunks.

Frontend (Start/Stop control + state badge) lands in the next push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

backend/app/services/gallery_dl.py

@@ -56,25 +56,24 @@ class ErrorType(StrEnum):
 # 20 contiguous HEADs is still negligible.
 TICK_SKIP_VALUE = "exit:20"
 
-# Backfill mode (operator-triggered deep scan): walk the full history.
-# Source.backfill_runs_remaining > 0 selects this mode; the longer
-# timeout below absorbs creators with thousands of posts.
+# Backfill mode (operator-triggered deep scan): walk the full history,
+# one TIME-BOXED CHUNK per run (plan #693). config_overrides["_backfill_state"]
+# == "running" selects this mode; the cursor checkpoint (plan #689) lets each
+# chunk resume where the last stopped, so the walk advances across chunks
+# until gallery-dl exits cleanly (= reached the bottom).
 #
-# Sits below download_source's Celery soft_time_limit
-# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py) with ~180s of
-# headroom for phase-3 persist. subprocess.run MUST raise TimeoutExpired
-# before Celery raises SoftTimeLimitExceeded — that exception path
-# captures partial stdout/stderr and finalizes the event; the soft-limit
-# path (until the 2026-06-03 fix) did not. Audit history: 1800 guaranteed
-# SIGKILL against the old hard limit (Knuxy #38275); 1170 was then sized
-# "30s shy of the hard limit (1200)" but still EXCEEDED the soft limit
-# (900), so SoftTimeLimitExceeded preempted TimeoutExpired and every
-# backfill stranded empty (Anduo #39912). Raising the Celery soft/hard
-# limits to 1350/1500 (tasks/download.py) is what made 1170 safe.
-# backfill_runs_remaining=3 still gives ~58 minutes of cumulative walk
-# across three runs for prolific creators.
+# The chunk budget is deliberately FAR below download_source's Celery
+# soft_time_limit (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py), not
+# "just under" it. Hitting this budget is the NORMAL chunk boundary, not a
+# failure: subprocess.run raises TimeoutExpired (which captures partial
+# stdout/stderr + the last emitted cursor), and download_service reclassifies
+# a chunk that made progress as PARTIAL (status "ok"), not an error. The huge
+# headroom means a stuck file or a slow chunk can never let Celery's
+# SoftTimeLimitExceeded preempt TimeoutExpired (the failure mode behind Knuxy
+# #38275 / Anduo #39912/#40411). Earlier we ran one ~1170s run-to-the-wall
+# per arming; that died as a timeout error every time on large catalogs.
 BACKFILL_SKIP_VALUE = True
-BACKFILL_TIMEOUT_SECONDS = 1170
+BACKFILL_CHUNK_SECONDS = 600
 
 
 # Sits well below download_source's Celery soft_time_limit
@@ -281,7 +280,12 @@ class GalleryDLService:
                 "skip": True,
                 "sleep": self._rate_limit,
                 "sleep-request": max(0.5, self._rate_limit / 4),
-                "retries": 3,
+                # 2 (not 3) retries — a stuck CDN host shouldn't burn a whole
+                # backfill chunk on one file. Anduo #40838 spent ~600s on a
+                # single image (4 extractor HEAD retries × 30s + 4 downloader
+                # GET retries × 120s); halving retries + the downloader
+                # timeout below caps a wedged file at ~1-2 min instead.
+                "retries": 2,
                 "timeout": 30.0,
                 "verify": True,
                 "postprocessors": [
@@ -296,8 +300,12 @@ class GalleryDLService:
             "downloader": {
                 "part": True,
                 "part-directory": str(self._config_dir / "temp"),
-                "retries": 3,
-                "timeout": 120.0,
+                # See the extractor retries note above (Anduo #40838): 2
+                # retries + a 60s read-timeout (was 120) so a stalled
+                # connection fails fast. 60s is a per-read timeout, not a
+                # transfer cap — large GIFs that keep streaming are unaffected.
+                "retries": 2,
+                "timeout": 60.0,
                 # Forward Patreon as Referer/Origin to yt-dlp when it
                 # fetches video manifests. Operator-flagged 2026-06-01
                 # (DaferQ patreon): video posts hosted on Mux carry a JWT