agent: stop downloader pool stampeding a slow curator (congestion collapse) #177

2026-07-01T10:45:35-04:00

bvandeusen commented

2026-07-01 10:45:35 -04:00

Fixes the operator-hit outage: agent looped on "curator unreachable" while curator's API (lease) was fine and the browser could still load images, just slowly. Root cause is a feedback loop in the new pipeline — every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS when the buffer is empty. A slow/failing curator ALSO empties the buffer, so the agent piled on more concurrent large-file GETs → saturated curator web + NFS → slower → more failures → more downloaders. Congestion collapse.

Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing; ramps back only once downloads succeed.
DL_MAX 24 → 8: 24 concurrent large-file downloads through one serving path is too many.
fetch_image timeout 180 → (10, 60): between-bytes read timeout, so a large-but-flowing download still completes but a stuck connection fails in 60s instead of hanging a downloader for 3 min.

Build marker 2026-07-01.4. CI green on dev (run 1821). See issue #1223.

🤖 Generated with Claude Code

Fixes the operator-hit outage: agent looped on "curator unreachable" while curator's API (lease) was fine and the browser could still load images, just slowly. Root cause is a feedback loop in the new pipeline — every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS when the buffer is empty. A slow/failing curator ALSO empties the buffer, so the agent piled on more concurrent large-file GETs → saturated curator web + NFS → slower → more failures → more downloaders. Congestion collapse. - **Failure-aware autoscaling**: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing; ramps back only once downloads succeed. - **DL_MAX 24 → 8**: 24 concurrent large-file downloads through one serving path is too many. - **fetch_image timeout 180 → (10, 60)**: between-bytes read timeout, so a large-but-flowing download still completes but a stuck connection fails in 60s instead of hanging a downloader for 3 min. Build marker `2026-07-01.4`. CI green on dev (run 1821). See issue #1223. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 1 commit 2026-07-01 10:45:35 -04:00

fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse)

CI / lint (push) Successful in 3s

Details

CI / frontend-build (push) Successful in 21s

Details

CI / backend-lint-and-test (push) Successful in 27s

Details

CI / integration (push) Successful in 3m26s

Details

ccbb5cbc9e

Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.

- Failure-aware autoscaling: if transient download failures rose since the last
  decision, SHRINK the downloader pool toward the floor instead of growing — the
  empty buffer is caused by failures, not the GPU starving. It ramps back up only
  once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
  path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
  large-but-flowing download still completes, but a stuck/dead connection fails in
  60s instead of hanging a downloader for 3 min and piling up stuck requests.

Build marker 2026-07-01.4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

bvandeusen merged commit 7ddad231f8 into main

2026-07-01 10:45:48 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#177