Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.
- Failure-aware autoscaling: if transient download failures rose since the last
decision, SHRINK the downloader pool toward the floor instead of growing — the
empty buffer is caused by failures, not the GPU starving. It ramps back up only
once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
large-but-flowing download still completes, but a stuck/dead connection fails in
60s instead of hanging a downloader for 3 min and piling up stuck requests.
Build marker 2026-07-01.4.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
bvandeusen
ccbb5cbc9e