fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s

Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.

- Failure-aware autoscaling: if transient download failures rose since the last
  decision, SHRINK the downloader pool toward the floor instead of growing — the
  empty buffer is caused by failures, not the GPU starving. It ramps back up only
  once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
  path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
  large-but-flowing download still completes, but a stuck/dead connection fails in
  60s instead of hanging a downloader for 3 min and piling up stuck requests.

Build marker 2026-07-01.4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-07-01 10:33:18 -04:00
parent ef3318aac1
commit ccbb5cbc9e
3 changed files with 34 additions and 8 deletions
+5 -1
View File
@@ -113,7 +113,11 @@ class FcClient:
def fetch_image(self, image_url: str) -> bytes:
# image_url is a server-relative path ("/images/...").
r = self.s.get(f"{self.base}{image_url}", timeout=180)
# timeout=(connect, read): the read timeout is BETWEEN-BYTES, not total,
# so a large-but-flowing download still completes — but a stuck/dead
# connection (curator overloaded) fails in 60s instead of hanging a
# downloader for 180s and piling up concurrent stuck requests on curator.
r = self.s.get(f"{self.base}{image_url}", timeout=(10, 60))
r.raise_for_status()
return r.content