fix(download): salvage soft-time-limit kills + fix timeout ladder
Backfill downloads stranded with empty logs + a generic "stranded by recovery sweep" error. Root cause: the backfill gallery-dl subprocess timeout (1170s) exceeded download_source's Celery soft_time_limit (900s), so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The TimeoutExpired path (which captures partial stdout/stderr and finalizes the event) never ran, the event was left 'running', and phase 3 never decremented backfill_runs_remaining — so the source re-ran and re-stranded every tick (Anduo #39912). Two layers: 1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both subprocess budgets (870 tick / 1170 backfill) sit below the soft limit with phase-3 persist headroom. Promote to module constants and guard the invariant with a test. 2. Catch SoftTimeLimitExceeded in download_source and finalize the in-flight event with a real reason, mirror phase-3 source-health, and decrement backfill so a chronically-slow source self-heals to tick mode. The existing celery_signals handler only covered TaskRun, not DownloadEvent — that was the gap. Updates stale 900/1200 references in gallery_dl.py + maintenance.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -59,29 +59,33 @@ TICK_SKIP_VALUE = "exit:20"
|
||||
# Source.backfill_runs_remaining > 0 selects this mode; the longer
|
||||
# timeout below absorbs creators with thousands of posts.
|
||||
#
|
||||
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
|
||||
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
|
||||
# before Celery SIGKILLs the worker — same rationale as the tick
|
||||
# default at line 74. The audit (2026-06-02) caught this at 1800,
|
||||
# guaranteeing SIGKILL on any backfill that ran to its subprocess
|
||||
# budget: stdout/stderr lost, backfill_runs_remaining never
|
||||
# decrements, recovery sweep stamps generic "stranded" 30 min later.
|
||||
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
|
||||
# was added to prevent. backfill_runs_remaining=3 still gives ~58
|
||||
# minutes of cumulative walk across three runs for prolific creators.
|
||||
# Sits below download_source's Celery soft_time_limit
|
||||
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py) with ~180s of
|
||||
# headroom for phase-3 persist. subprocess.run MUST raise TimeoutExpired
|
||||
# before Celery raises SoftTimeLimitExceeded — that exception path
|
||||
# captures partial stdout/stderr and finalizes the event; the soft-limit
|
||||
# path (until the 2026-06-03 fix) did not. Audit history: 1800 guaranteed
|
||||
# SIGKILL against the old hard limit (Knuxy #38275); 1170 was then sized
|
||||
# "30s shy of the hard limit (1200)" but still EXCEEDED the soft limit
|
||||
# (900), so SoftTimeLimitExceeded preempted TimeoutExpired and every
|
||||
# backfill stranded empty (Anduo #39912). Raising the Celery soft/hard
|
||||
# limits to 1350/1500 (tasks/download.py) is what made 1170 safe.
|
||||
# backfill_runs_remaining=3 still gives ~58 minutes of cumulative walk
|
||||
# across three runs for prolific creators.
|
||||
BACKFILL_SKIP_VALUE = True
|
||||
BACKFILL_TIMEOUT_SECONDS = 1170
|
||||
|
||||
|
||||
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
|
||||
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
|
||||
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
|
||||
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
|
||||
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
|
||||
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
|
||||
# The 30s buffer absorbs scheduler jitter / GC pauses without making
|
||||
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
|
||||
# still live in source.config_overrides for legitimately long syncs.
|
||||
# Sits well below download_source's Celery soft_time_limit
|
||||
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py). subprocess.run MUST
|
||||
# raise TimeoutExpired before Celery raises SoftTimeLimitExceeded —
|
||||
# otherwise Celery wins the race, SIGKILLs the worker, in-memory
|
||||
# stdout/stderr is lost, and the DownloadEvent ends up empty-logged with
|
||||
# "stranded by recovery sweep" (operator-flagged 2026-05-31, Knuxy event
|
||||
# #38275; recurred in backfill mode as Anduo #39912). Per-source bumps
|
||||
# still live in source.config_overrides for legitimately long syncs —
|
||||
# keep any override below the soft limit, or the soft-limit salvage path
|
||||
# in tasks/download.py (_finalize_soft_limited) is the only safety net.
|
||||
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user