feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+13
-14
@@ -122,26 +122,25 @@ async def delete_source(source_id: int):
|
||||
|
||||
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
|
||||
async def set_backfill(source_id: int):
|
||||
"""Plan #544: arm a source for backfill mode for the next N download
|
||||
runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
|
||||
source dict. While backfill_runs_remaining > 0, downloads use
|
||||
gallery-dl's full-walk config (skip: True + 30-min timeout) instead
|
||||
of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
|
||||
"""Plan #693: start or stop a run-until-done backfill. Body:
|
||||
`{"action": "start" | "stop"}` (default "start"). 'start' walks the full
|
||||
post history in time-boxed chunks until it reaches the bottom (then the
|
||||
source shows 'complete'); 'stop' cancels back to tick mode. Returns the
|
||||
updated source dict (incl. backfill_state / backfill_chunks)."""
|
||||
payload = await request.get_json(silent=True) or {}
|
||||
runs = payload.get("runs", 3)
|
||||
try:
|
||||
runs = int(runs)
|
||||
except (TypeError, ValueError):
|
||||
return _bad("invalid_runs", detail="runs must be an integer")
|
||||
action = payload.get("action", "start")
|
||||
if action not in ("start", "stop"):
|
||||
return _bad("invalid_action", detail="action must be 'start' or 'stop'")
|
||||
async with get_session() as session:
|
||||
try:
|
||||
record = await SourceService(session).set_backfill_runs(
|
||||
source_id, runs,
|
||||
svc = SourceService(session)
|
||||
record = (
|
||||
await svc.start_backfill(source_id)
|
||||
if action == "start"
|
||||
else await svc.stop_backfill(source_id)
|
||||
)
|
||||
except LookupError:
|
||||
return _bad("not_found", status=404)
|
||||
except ValueError as exc:
|
||||
return _bad("invalid_runs", detail=str(exc))
|
||||
return jsonify(record.to_dict())
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user