fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Failing after 3m19s

The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).

Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
  (connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
  across chunks in _stream_to_file (HTTP has no total-download timeout), and
  passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.

Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.

Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.

Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 21:15:49 -04:00
parent 25e1e098fb
commit 4272a19d40
5 changed files with 129 additions and 41 deletions
+2 -2
View File
@@ -81,7 +81,7 @@ def test_fetch_external_link_downloads_and_attaches(db_sync, tmp_path, monkeypat
monkeypatch.setattr(ext, "IMAGES_ROOT", tmp_path)
monkeypatch.setattr(ext, "_redis", lambda: _FakeRedis())
def fake_fetch(host, url, dest_dir, *, timeout, should_stop=lambda: False):
def fake_fetch(host, url, dest_dir, **kwargs):
dest_dir.mkdir(parents=True, exist_ok=True)
f = dest_dir / "film.bin" # non-art → PostAttachment (no thumb/ML enqueue)
f.write_bytes(b"a film pack")
@@ -115,7 +115,7 @@ def test_refetch_same_link_keeps_canonical_file(db_sync, tmp_path, monkeypatch):
monkeypatch.setattr(ext, "IMAGES_ROOT", tmp_path)
monkeypatch.setattr(ext, "_redis", lambda: _FakeRedis())
def fake_fetch(host, url, dest_dir, *, timeout, should_stop=lambda: False):
def fake_fetch(host, url, dest_dir, **kwargs):
dest_dir.mkdir(parents=True, exist_ok=True)
f = dest_dir / "clip.jpg" # art → ImageRecord (imported in place)
f.write_bytes(_structured_jpeg_bytes())