fix(agent): stream videos via ffmpeg-from-URL instead of downloading the whole file
The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE file into memory (r.content) just to sample a few frames, which buffered ~1GB in RAM and — on any slow/contended media store — got cut off mid-download (ChunkedEncodingError), failed, and re-leased forever. Measured the media read at ~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak helps; the file simply shouldn't be fully downloaded. Environment-agnostic fix (works for any deployment, completes even when slow): - media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL. It Range-reads only the video index + up to max_frames of content — never the whole file — and reconnect flags resume a dropped transfer instead of failing. Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over speed. Removes the bytes-based sample_frames (dead once videos stream). - worker._download_decode: videos now stream (no fetch_image, no RAM blowup); stills still download+decode. On an ffmpeg miss, probe curator liveness (client.is_reachable) → fail the job if curator is up (unprocessable file, stops the infinite re-lease) vs release if curator is down (transient, survives a redeploy). Auth header passed so it works whether or not /images is gated. Build marker 2026-07-01.6. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -121,6 +121,16 @@ class FcClient:
|
||||
r.raise_for_status()
|
||||
return r.content
|
||||
|
||||
def is_reachable(self) -> bool:
|
||||
"""Cheap 'is curator responding at all right now?' check. Used to decide,
|
||||
when a video can't be sampled, between a transient outage (keep retrying —
|
||||
survives a redeploy) and an unprocessable file (fail it, don't loop)."""
|
||||
try:
|
||||
r = self.s.get(f"{self.base}/api/gpu/status", timeout=5)
|
||||
return r.status_code < 500
|
||||
except requests.RequestException:
|
||||
return False
|
||||
|
||||
def queue_status(self) -> dict:
|
||||
# Short timeout: this backs the UI /status poll, so a busy curator must
|
||||
# not hang the page for long (the GPU meters poll /gpu separately).
|
||||
|
||||
Reference in New Issue
Block a user