fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)

The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).

Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
  (connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
  across chunks in _stream_to_file (HTTP has no total-download timeout), and
  passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.

Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.

Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.

Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/app/tasks/external.py

@@ -41,16 +41,19 @@ IMAGES_ROOT = Path("/images")
 # After this many failed attempts a link is dead-lettered (skipped by routine
 # sweeps; an operator recovery still re-attempts). Mirrors the ingester ledger.
 DEAD_LETTER_THRESHOLD = 3
-# Per-fetch wall-clock budget — films/packs are large; the celery time_limit is
-# the backstop above this.
-_FETCH_TIMEOUT = 3000.0
+# Per-fetch read + total budgets now live in external_fetch (a short read
+# timeout fails a stalled host fast; a generous total caps a big-but-flowing
+# download). The celery soft/hard time_limit is the outer backstop above those.
 # Links enqueued per sweep — bounds the burst when a big backfill records many.
 _SWEEP_BATCH = 50
 # Dead rows older than this are pruned (retention).
 _RETENTION_DAYS = 30
-# Per-host serialize lock.
+# Per-host serialize lock. TTL is a safety net for a worker that dies holding it
+# (normal completion/error releases it in `finally`); sized just past the fetch
+# total budget (30 min) so a dead worker can't wedge a host's links much longer
+# than one fetch would have taken.
 _LOCK_PREFIX = "fc:extdl_lock:"
-_LOCK_TTL = 3600
+_LOCK_TTL = 2400
 _SERIALIZE_COUNTDOWN = 120
 _MAX_SERIALIZE_WAITS = 30
 
@@ -177,7 +180,7 @@ def fetch_external_link(self, link_id: int, _serialize_waits: int = 0) -> dict:
         / str(post.external_post_id) / str(link_id)
     )
     try:
-        result = fetch_external(host, url, post_dir, timeout=_FETCH_TIMEOUT)
+        result = fetch_external(host, url, post_dir)
         with SessionLocal() as session:
             link = session.get(ExternalLink, link_id)
             if not result.ok:

backend/app/tasks/maintenance.py

@@ -147,13 +147,14 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
     "backend.app.tasks.backup.restore_images_task": 420,
     # Library audit scans the full library — 2h hard limit.
     "backend.app.tasks.library_audit.scan_library_for_rule": 130,
-    # External file-host fetches (mega/gdrive/film packs) legitimately run to
-    # the task's 60-min hard limit (time_limit=3600; per-fetch _FETCH_TIMEOUT
-    # is 50min). Its TaskRun records queue='default' (no queue override), so
-    # without this it fell to the 5-min default and healthy in-flight fetches
-    # were phantom-flagged 'RecoverySweep' before their own timeout/error could
+    # External file-host fetches (mega/gdrive/film packs) can run to the task's
+    # 60-min hard limit (time_limit=3600) — the fetcher's own read/total budgets
+    # (external_fetch) cap a single fetch below that, but this stays the outer
+    # backstop. Without an override these healthy in-flight fetches were
+    # phantom-flagged 'RecoverySweep' before their own timeout/error could
     # surface (operator-flagged 2026-06-17, target 414 swept at 6.6min). A
-    # task-name override is robust whatever queue the row records. 65 = 60 + 5.
+    # task-name override beats the queue threshold whatever queue the row records
+    # (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
     "backend.app.tasks.external.fetch_external_link": 65,
 }