fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse)
Operator hit an outage after the machine slept overnight: the agent showed "curator unreachable" in a loop while curator's API (lease) was actually fine and the browser could still load images — just slowly. Root cause is a feedback loop in the new pipeline: every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for that reason — so the agent piled on more concurrent large-file GETs, saturating curator's web workers + NFS, which slowed curator (and its browser) further and produced more failures → more downloaders. Classic congestion collapse. - Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing — the empty buffer is caused by failures, not the GPU starving. It ramps back up only once downloads succeed again. - DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving path is too many; 8 keeps a fast GPU fed without stampeding curator. - fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a large-but-flowing download still completes, but a stuck/dead connection fails in 60s instead of hanging a downloader for 3 min and piling up stuck requests. Build marker 2026-07-01.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -113,7 +113,11 @@ class FcClient:
|
||||
|
||||
def fetch_image(self, image_url: str) -> bytes:
|
||||
# image_url is a server-relative path ("/images/...").
|
||||
r = self.s.get(f"{self.base}{image_url}", timeout=180)
|
||||
# timeout=(connect, read): the read timeout is BETWEEN-BYTES, not total,
|
||||
# so a large-but-flowing download still completes — but a stuck/dead
|
||||
# connection (curator overloaded) fails in 60s instead of hanging a
|
||||
# downloader for 180s and piling up concurrent stuck requests on curator.
|
||||
r = self.s.get(f"{self.base}{image_url}", timeout=(10, 60))
|
||||
r.raise_for_status()
|
||||
return r.content
|
||||
|
||||
|
||||
Reference in New Issue
Block a user