fix(download): salvage soft-time-limit kills + fix timeout ladder
CI / lint (push) Failing after 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 21s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m22s
CI / intcore (push) Successful in 8m4s

Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).

Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
   subprocess budgets (870 tick / 1170 backfill) sit below the soft
   limit with phase-3 persist headroom. Promote to module constants and
   guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
   in-flight event with a real reason, mirror phase-3 source-health, and
   decrement backfill so a chronically-slow source self-heals to tick
   mode. The existing celery_signals handler only covered TaskRun, not
   DownloadEvent — that was the gap.

Updates stale 900/1200 references in gallery_dl.py + maintenance.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-03 16:56:13 -04:00
parent 3162cff96b
commit 6590dcdb39
4 changed files with 294 additions and 30 deletions
+23 -19
View File
@@ -59,29 +59,33 @@ TICK_SKIP_VALUE = "exit:20"
# Source.backfill_runs_remaining > 0 selects this mode; the longer
# timeout below absorbs creators with thousands of posts.
#
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# before Celery SIGKILLs the worker — same rationale as the tick
# default at line 74. The audit (2026-06-02) caught this at 1800,
# guaranteeing SIGKILL on any backfill that ran to its subprocess
# budget: stdout/stderr lost, backfill_runs_remaining never
# decrements, recovery sweep stamps generic "stranded" 30 min later.
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
# was added to prevent. backfill_runs_remaining=3 still gives ~58
# minutes of cumulative walk across three runs for prolific creators.
# Sits below download_source's Celery soft_time_limit
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py) with ~180s of
# headroom for phase-3 persist. subprocess.run MUST raise TimeoutExpired
# before Celery raises SoftTimeLimitExceeded — that exception path
# captures partial stdout/stderr and finalizes the event; the soft-limit
# path (until the 2026-06-03 fix) did not. Audit history: 1800 guaranteed
# SIGKILL against the old hard limit (Knuxy #38275); 1170 was then sized
# "30s shy of the hard limit (1200)" but still EXCEEDED the soft limit
# (900), so SoftTimeLimitExceeded preempted TimeoutExpired and every
# backfill stranded empty (Anduo #39912). Raising the Celery soft/hard
# limits to 1350/1500 (tasks/download.py) is what made 1170 safe.
# backfill_runs_remaining=3 still gives ~58 minutes of cumulative walk
# across three runs for prolific creators.
BACKFILL_SKIP_VALUE = True
BACKFILL_TIMEOUT_SECONDS = 1170
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
# The 30s buffer absorbs scheduler jitter / GC pauses without making
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
# still live in source.config_overrides for legitimately long syncs.
# Sits well below download_source's Celery soft_time_limit
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py). subprocess.run MUST
# raise TimeoutExpired before Celery raises SoftTimeLimitExceeded —
# otherwise Celery wins the race, SIGKILLs the worker, in-memory
# stdout/stderr is lost, and the DownloadEvent ends up empty-logged with
# "stranded by recovery sweep" (operator-flagged 2026-05-31, Knuxy event
# #38275; recurred in backfill mode as Anduo #39912). Per-source bumps
# still live in source.config_overrides for legitimately long syncs —
# keep any override below the soft limit, or the soft-limit salvage path
# in tasks/download.py (_finalize_soft_limited) is the only safety net.
_DEFAULT_GDL_TIMEOUT_SECONDS = 870