fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -81,7 +81,7 @@ def test_fetch_external_link_downloads_and_attaches(db_sync, tmp_path, monkeypat
|
||||
monkeypatch.setattr(ext, "IMAGES_ROOT", tmp_path)
|
||||
monkeypatch.setattr(ext, "_redis", lambda: _FakeRedis())
|
||||
|
||||
def fake_fetch(host, url, dest_dir, *, timeout, should_stop=lambda: False):
|
||||
def fake_fetch(host, url, dest_dir, **kwargs):
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
f = dest_dir / "film.bin" # non-art → PostAttachment (no thumb/ML enqueue)
|
||||
f.write_bytes(b"a film pack")
|
||||
@@ -115,7 +115,7 @@ def test_refetch_same_link_keeps_canonical_file(db_sync, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(ext, "IMAGES_ROOT", tmp_path)
|
||||
monkeypatch.setattr(ext, "_redis", lambda: _FakeRedis())
|
||||
|
||||
def fake_fetch(host, url, dest_dir, *, timeout, should_stop=lambda: False):
|
||||
def fake_fetch(host, url, dest_dir, **kwargs):
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
f = dest_dir / "clip.jpg" # art → ImageRecord (imported in place)
|
||||
f.write_bytes(_structured_jpeg_bytes())
|
||||
|
||||
Reference in New Issue
Block a user