Smarter backfill: time-boxed chunks, run-until-done (plan #693) #71

2026-06-05T16:32:50-04:00

bvandeusen commented

2026-06-05 16:32:50 -04:00

Replaces the fixed run-count backfill with a run-until-done state machine that walks an artist's full post history in time-boxed chunks.

Backend (`96fffaf`)

Time-boxed chunks: each chunk runs BACKFILL_CHUNK_SECONDS=600 (safely below the 1350s Celery soft limit), checkpoints its cursor, then yields.
Progress, not errors: a chunk timeout that made progress (cursor advanced or files landed) is reclassified TIMEOUT → PARTIAL → event status "ok", so chunks read as completed/progress instead of dying at the wall.
State machine in Source.config_overrides: running → complete (clean exit reaches the bottom of the post list) or stalled (200-chunk cap, or 2 consecutive no-advance chunks).
Retry tuning (Anduo #40838): extractor/downloader retries 3→2, downloader read-timeout 120→60s — caps a wedged file at ~1–2 min instead of ~600s. (Verified: this is an idle/read timeout, not a transfer cap — multi-minute archive downloads stream uninterrupted; large files that span chunks resume via part:True + cursor re-walk.)

Frontend (`618dafd`)

Backfill control toggles Start ⇄ Stop.
Chip becomes a state badge: "Backfilling (chunk N)" / "Backfilled" / "Stalled".

CI green on dev (run #574: lint, backend unit, frontend build, integration all ✓).

🤖 Generated with Claude Code

Replaces the fixed run-count backfill with a run-until-done state machine that walks an artist's full post history in time-boxed chunks. ## Backend (`96fffaf`) - **Time-boxed chunks**: each chunk runs `BACKFILL_CHUNK_SECONDS=600` (safely below the 1350s Celery soft limit), checkpoints its cursor, then yields. - **Progress, not errors**: a chunk timeout that made progress (cursor advanced or files landed) is reclassified `TIMEOUT → PARTIAL` → event status "ok", so chunks read as completed/progress instead of dying at the wall. - **State machine** in `Source.config_overrides`: `running` → `complete` (clean exit reaches the bottom of the post list) or `stalled` (200-chunk cap, or 2 consecutive no-advance chunks). - **Retry tuning** (Anduo #40838): extractor/downloader `retries 3→2`, downloader read-timeout `120→60s` — caps a wedged file at ~1–2 min instead of ~600s. (Verified: this is an idle/read timeout, not a transfer cap — multi-minute archive downloads stream uninterrupted; large files that span chunks resume via `part:True` + cursor re-walk.) ## Frontend (`618dafd`) - Backfill control toggles Start ⇄ Stop. - Chip becomes a state badge: "Backfilling (chunk N)" / "Backfilled" / "Stalled". CI green on dev (run #574: lint, backend unit, frontend build, integration all ✓). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-05 16:32:50 -04:00

feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)

CI / lint (push) Successful in 2s

Details

CI / backend-lint-and-test (push) Successful in 11s

Details

CI / frontend-build (push) Successful in 17s

Details

CI / integration (push) Successful in 2m58s

Details

96fffaff64

Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).

- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
  far under the 1350 soft limit. Hitting it = normal chunk boundary (the
  TimeoutExpired path already captures partial output + the cursor), not a
  near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
  (running/complete/stalled). A running backfill auto-continues in chunks
  across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
  'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
  pause a pathological walk as 'stalled'. Replaces the N-runs counter
  (backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
  and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
  one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
  start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
  source dict exposes backfill_state + backfill_chunks.

Frontend (Start/Stop control + state badge) lands in the next push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(subscriptions): smarter-backfill UI — Start/Stop + state badge

CI / lint (push) Successful in 2s

Details

CI / backend-lint-and-test (push) Successful in 11s

Details

CI / frontend-build (push) Successful in 18s

Details

CI / integration (push) Successful in 2m58s

Details

618dafde85

Plan #693 (frontend). Backend landed in 96fffaf.

- sources store: setBackfill(runs) → startBackfill/stopBackfill ({action}).
- SubscriptionsTab: the deep-scan window.prompt for N runs becomes a
  Start/Stop toggle keyed on source.backfill_state.
- SourceRow + SourceCard: the 'backfill (N×)' chip becomes a state badge —
  Backfilling (chunk N) / Backfilled / Stalled — and the scan button
  toggles between Backfill (mdi-magnify-scan) and Stop (mdi-stop).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit 7395e77d75 into main

2026-06-05 16:33:07 -04:00

bvandeusen referenced this issue from a commit

2026-06-17 12:03:48 -04:00

feat(subscribestar): flip dispatch to the native ingester (#893, Step 5)

bvandeusen referenced this pull request

2026-06-17 12:48:20 -04:00

SubscribeStar → native core ingester + native-ingest DRY pass (milestone #71) #117

bvandeusen referenced this issue from a commit

2026-06-17 12:48:31 -04:00

Merge pull request 'SubscribeStar → native core ingester + native-ingest DRY pass (milestone #71)' (#117) from dev into main

Sign in to join this conversation.