fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -147,13 +147,14 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
"backend.app.tasks.backup.restore_images_task": 420,
|
||||
# Library audit scans the full library — 2h hard limit.
|
||||
"backend.app.tasks.library_audit.scan_library_for_rule": 130,
|
||||
# External file-host fetches (mega/gdrive/film packs) legitimately run to
|
||||
# the task's 60-min hard limit (time_limit=3600; per-fetch _FETCH_TIMEOUT
|
||||
# is 50min). Its TaskRun records queue='default' (no queue override), so
|
||||
# without this it fell to the 5-min default and healthy in-flight fetches
|
||||
# were phantom-flagged 'RecoverySweep' before their own timeout/error could
|
||||
# External file-host fetches (mega/gdrive/film packs) can run to the task's
|
||||
# 60-min hard limit (time_limit=3600) — the fetcher's own read/total budgets
|
||||
# (external_fetch) cap a single fetch below that, but this stays the outer
|
||||
# backstop. Without an override these healthy in-flight fetches were
|
||||
# phantom-flagged 'RecoverySweep' before their own timeout/error could
|
||||
# surface (operator-flagged 2026-06-17, target 414 swept at 6.6min). A
|
||||
# task-name override is robust whatever queue the row records. 65 = 60 + 5.
|
||||
# task-name override beats the queue threshold whatever queue the row records
|
||||
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
||||
"backend.app.tasks.external.fetch_external_link": 65,
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user