fix(agent): stream videos via ffmpeg-from-URL instead of downloading the whole file
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s

The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE
file into memory (r.content) just to sample a few frames, which buffered ~1GB in
RAM and — on any slow/contended media store — got cut off mid-download
(ChunkedEncodingError), failed, and re-leased forever. Measured the media read at
~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak
helps; the file simply shouldn't be fully downloaded.

Environment-agnostic fix (works for any deployment, completes even when slow):
- media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL.
  It Range-reads only the video index + up to max_frames of content — never the
  whole file — and reconnect flags resume a dropped transfer instead of failing.
  Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over
  speed. Removes the bytes-based sample_frames (dead once videos stream).
- worker._download_decode: videos now stream (no fetch_image, no RAM blowup);
  stills still download+decode. On an ffmpeg miss, probe curator liveness
  (client.is_reachable) → fail the job if curator is up (unprocessable file, stops
  the infinite re-lease) vs release if curator is down (transient, survives a
  redeploy). Auth header passed so it works whether or not /images is gated.

Build marker 2026-07-01.6. Refs issue #1225.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-07-01 14:15:22 -04:00
parent aa0605585b
commit 8a0237eeea
5 changed files with 82 additions and 32 deletions
+3
View File
@@ -41,6 +41,8 @@ class Config:
frame_dedupe_distance: int # video frames whose dHash differs by < this many
# bits are near-duplicates, dropped before detect;
# higher keeps more frames, 0 disables
ffmpeg_timeout: float # hard ceiling (s) for ffmpeg-from-URL video sampling;
# generous so a SLOW media link still completes
@classmethod
def from_env(cls) -> Config:
@@ -69,4 +71,5 @@ class Config:
max_regions=int(os.environ.get("MAX_REGIONS", "128")),
dedupe_iou=float(os.environ.get("DEDUPE_IOU", "0.85")),
frame_dedupe_distance=int(os.environ.get("FRAME_DEDUPE_DISTANCE", "8")),
ffmpeg_timeout=float(os.environ.get("FFMPEG_TIMEOUT", "1200")),
)