Smarter backfill: time-boxed chunks, run-until-done (plan #693) #71

Merged
bvandeusen merged 2 commits from dev into main 2026-06-05 16:33:07 -04:00

2 Commits

Author SHA1 Message Date
bvandeusen 618dafde85 feat(subscriptions): smarter-backfill UI — Start/Stop + state badge
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 18s
CI / integration (push) Successful in 2m58s
Plan #693 (frontend). Backend landed in 96fffaf.

- sources store: setBackfill(runs) → startBackfill/stopBackfill ({action}).
- SubscriptionsTab: the deep-scan window.prompt for N runs becomes a
  Start/Stop toggle keyed on source.backfill_state.
- SourceRow + SourceCard: the 'backfill (N×)' chip becomes a state badge —
  Backfilling (chunk N) / Backfilled / Stalled — and the scan button
  toggles between Backfill (mdi-magnify-scan) and Stop (mdi-stop).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 15:06:30 -04:00
bvandeusen 96fffaff64 feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / integration (push) Successful in 2m58s
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).

- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
  far under the 1350 soft limit. Hitting it = normal chunk boundary (the
  TimeoutExpired path already captures partial output + the cursor), not a
  near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
  (running/complete/stalled). A running backfill auto-continues in chunks
  across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
  'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
  pause a pathological walk as 'stalled'. Replaces the N-runs counter
  (backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
  and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
  one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
  start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
  source dict exposes backfill_state + backfill_chunks.

Frontend (Start/Stop control + state badge) lands in the next push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 15:02:46 -04:00