Replaces the fixed run-count backfill with a run-until-done state machine that walks an artist's full post history in time-boxed chunks.
Backend (96fffaf)
Time-boxed chunks: each chunk runs BACKFILL_CHUNK_SECONDS=600 (safely below the 1350s Celery soft limit), checkpoints its cursor, then yields.
Progress, not errors: a chunk timeout that made progress (cursor advanced or files landed) is reclassified TIMEOUT → PARTIAL → event status "ok", so chunks read as completed/progress instead of dying at the wall.
State machine in Source.config_overrides: running → complete (clean exit reaches the bottom of the post list) or stalled (200-chunk cap, or 2 consecutive no-advance chunks).
Retry tuning (Anduo #40838): extractor/downloader retries 3→2, downloader read-timeout 120→60s — caps a wedged file at ~1–2 min instead of ~600s. (Verified: this is an idle/read timeout, not a transfer cap — multi-minute archive downloads stream uninterrupted; large files that span chunks resume via part:True + cursor re-walk.)
Frontend (618dafd)
Backfill control toggles Start ⇄ Stop.
Chip becomes a state badge: "Backfilling (chunk N)" / "Backfilled" / "Stalled".
CI green on dev (run #574: lint, backend unit, frontend build, integration all ✓).
Replaces the fixed run-count backfill with a run-until-done state machine that walks an artist's full post history in time-boxed chunks.
## Backend (`96fffaf`)
- **Time-boxed chunks**: each chunk runs `BACKFILL_CHUNK_SECONDS=600` (safely below the 1350s Celery soft limit), checkpoints its cursor, then yields.
- **Progress, not errors**: a chunk timeout that made progress (cursor advanced or files landed) is reclassified `TIMEOUT → PARTIAL` → event status "ok", so chunks read as completed/progress instead of dying at the wall.
- **State machine** in `Source.config_overrides`: `running` → `complete` (clean exit reaches the bottom of the post list) or `stalled` (200-chunk cap, or 2 consecutive no-advance chunks).
- **Retry tuning** (Anduo #40838): extractor/downloader `retries 3→2`, downloader read-timeout `120→60s` — caps a wedged file at ~1–2 min instead of ~600s. (Verified: this is an idle/read timeout, not a transfer cap — multi-minute archive downloads stream uninterrupted; large files that span chunks resume via `part:True` + cursor re-walk.)
## Frontend (`618dafd`)
- Backfill control toggles Start ⇄ Stop.
- Chip becomes a state badge: "Backfilling (chunk N)" / "Backfilled" / "Stalled".
CI green on dev (run #574: lint, backend unit, frontend build, integration all ✓).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).
- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
far under the 1350 soft limit. Hitting it = normal chunk boundary (the
TimeoutExpired path already captures partial output + the cursor), not a
near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
(running/complete/stalled). A running backfill auto-continues in chunks
across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
pause a pathological walk as 'stalled'. Replaces the N-runs counter
(backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
source dict exposes backfill_state + backfill_chunks.
Frontend (Start/Stop control + state badge) lands in the next push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan #693 (frontend). Backend landed in 96fffaf.
- sources store: setBackfill(runs) → startBackfill/stopBackfill ({action}).
- SubscriptionsTab: the deep-scan window.prompt for N runs becomes a
Start/Stop toggle keyed on source.backfill_state.
- SourceRow + SourceCard: the 'backfill (N×)' chip becomes a state badge —
Backfilling (chunk N) / Backfilled / Stalled — and the scan button
toggles between Backfill (mdi-magnify-scan) and Stop (mdi-stop).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Replaces the fixed run-count backfill with a run-until-done state machine that walks an artist's full post history in time-boxed chunks.
Backend (
96fffaf)BACKFILL_CHUNK_SECONDS=600(safely below the 1350s Celery soft limit), checkpoints its cursor, then yields.TIMEOUT → PARTIAL→ event status "ok", so chunks read as completed/progress instead of dying at the wall.Source.config_overrides:running→complete(clean exit reaches the bottom of the post list) orstalled(200-chunk cap, or 2 consecutive no-advance chunks).retries 3→2, downloader read-timeout120→60s— caps a wedged file at ~1–2 min instead of ~600s. (Verified: this is an idle/read timeout, not a transfer cap — multi-minute archive downloads stream uninterrupted; large files that span chunks resume viapart:True+ cursor re-walk.)Frontend (
618dafd)CI green on dev (run #574: lint, backend unit, frontend build, integration all ✓).
🤖 Generated with Claude Code