fix(download): salvage soft-time-limit kills + fix timeout ladder #59

Merged

bvandeusen merged 2 commits from dev into main 2026-06-03 19:35:45 -04:00

Conversation 0 Commits 2 Files Changed 4

+294 -30

bvandeusen commented 2026-06-03 19:35:35 -04:00

Owner

Copy Link

Copy Source

Fixes backfill subscriptions stranding with empty logs + a generic "stranded by recovery sweep" error (operator-flagged 2026-06-03, patreon/Anduo #39912 → #39937).

Root cause

download_source ran gallery-dl in an executor thread with a backfill subprocess timeout of 1170s, but Celery's soft_time_limit was 900s. The soft-limit signal fires in the main thread, so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired — the path that captures partial stdout/stderr and finalizes the event never ran. The event was left running → recovery sweep stamped a context-free "stranded" 30 min later, and phase 3 never decremented backfill_runs_remaining, so the source re-ran and re-stranded every tick.

Changes

1. Fix the ladder — raise download_source limits to soft 1350 / hard 1500 so both subprocess budgets (870 tick / 1170 backfill) sit under the soft limit with phase-3 persist headroom. Promoted to module constants; invariant guarded by a test.
2. Defense in depth — download_source now catches SoftTimeLimitExceeded and finalizes the in-flight event with a real reason, mirrors phase-3 source-health, and decrements backfill so a chronically-slow source self-heals to tick mode. (The existing celery_signals handler only covered the TaskRun dashboard row, not the DownloadEvent — that was the gap.)
3. Corrected stale 900/1200 ladder comments in gallery_dl.py + maintenance.py.

Tests

4 new tests in tests/test_download_source_task.py (timeout-ladder invariant, decorated-limit match, soft-limit event salvage + backfill decrement, no-op-without-running-event). CI green on dev HEAD.

🤖 Generated with Claude Code

Fixes backfill subscriptions stranding with empty logs + a generic "stranded by recovery sweep" error (operator-flagged 2026-06-03, patreon/Anduo #39912 → #39937). ## Root cause `download_source` ran gallery-dl in an executor thread with a backfill subprocess timeout of **1170s**, but Celery's `soft_time_limit` was **900s**. The soft-limit signal fires in the main thread, so `SoftTimeLimitExceeded` preempted `subprocess.TimeoutExpired` — the path that captures partial stdout/stderr and finalizes the event never ran. The event was left `running` → recovery sweep stamped a context-free "stranded" 30 min later, and phase 3 never decremented `backfill_runs_remaining`, so the source re-ran and re-stranded every tick. ## Changes 1. **Fix the ladder** — raise `download_source` limits to soft **1350** / hard **1500** so both subprocess budgets (870 tick / 1170 backfill) sit under the soft limit with phase-3 persist headroom. Promoted to module constants; invariant guarded by a test. 2. **Defense in depth** — `download_source` now catches `SoftTimeLimitExceeded` and finalizes the in-flight event with a real reason, mirrors phase-3 source-health, and decrements backfill so a chronically-slow source self-heals to tick mode. (The existing `celery_signals` handler only covered the `TaskRun` dashboard row, not the `DownloadEvent` — that was the gap.) 3. Corrected stale 900/1200 ladder comments in `gallery_dl.py` + `maintenance.py`. ## Tests 4 new tests in `tests/test_download_source_task.py` (timeout-ladder invariant, decorated-limit match, soft-limit event salvage + backfill decrement, no-op-without-running-event). CI green on dev HEAD. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-03 19:35:35 -04:00

fix(download): salvage soft-time-limit kills + fix timeout ladder ... 6590dcdb39

Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).

Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
   subprocess budgets (870 tick / 1170 backfill) sit below the soft
   limit with phase-3 persist headroom. Promote to module constants and
   guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
   in-flight event with a real reason, mirror phase-3 source-health, and
   decrement backfill so a chronically-slow source self-heals to tick
   mode. The existing celery_signals handler only covered TaskRun, not
   DownloadEvent — that was the gap.

Updates stale 900/1200 references in gallery_dl.py + maintenance.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

style(test): fix ruff I001 import order in download task test ... 9cb24c9e1b

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit a8f6a464aa into main 2026-06-03 19:35:45 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#59