fix(agent): server-side rate metrics + killable-on-stop ffmpeg
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m25s

Two follow-ups from live debugging of "work/min never populates" and
"stopped never reached".

1) jobs/min + downloads/min are now computed in the BACKEND on a fixed
   cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were
   derived client-side from poll deltas with a dt<30s guard — but a
   backgrounded/unfocused browser tab throttles its timers to ~1/min, so
   every delta exceeded 30s and the guard blanked the rates forever. A
   server-side rate is independent of how often the tab polls. Frontend just
   displays s.jobs_per_min / s.downloads_per_min. VERSION → .9.

2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a
   slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see
   the stop signal until ffmpeg returned, so Stop detached still-running
   threads and work kept flowing long after — "stopped" that wasn't really
   stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a
   `should_stop` callback every 0.5s, terminating (then killing) the process
   at once on Stop or the per-video timeout. A stop-killed job is handed back
   (transient), not failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-01 16:51:26 -04:00
parent 6282e753a9
commit 713a11e394
3 changed files with 91 additions and 40 deletions
+39 -9
View File
@@ -5,6 +5,7 @@ import io
import os
import subprocess
import tempfile
import time
from PIL import Image, ImageFile
@@ -105,28 +106,57 @@ def _collect_frames(
return out
def _terminate(proc: subprocess.Popen) -> None:
"""Stop an ffmpeg cleanly, then hard-kill if it ignores SIGTERM."""
proc.terminate()
try:
proc.wait(timeout=2)
except subprocess.TimeoutExpired:
proc.kill()
try:
proc.wait(timeout=2)
except subprocess.TimeoutExpired:
pass
def sample_frames_from_url(
url: str, interval_seconds: float, max_frames: int,
*, headers: str = "", timeout: float = 1200.0,
*, headers: str = "", timeout: float = 1200.0, should_stop=None,
) -> list[tuple[float, Image.Image]]:
"""Sample frames by pointing ffmpeg STRAIGHT at the media URL — it Range-reads
only the video index + up to max_frames worth of content, so the agent never
downloads the whole file (VR/4K originals run 800MB+ and would buffer ~1GB in
RAM and get cut off mid-download). Reconnect flags resume a dropped transfer;
the generous timeout lets it finish however slow the link is — completion over
speed, and environment-agnostic (any HTTP+Range source). Empty on failure."""
the timeout is the per-video ceiling (a slow/reconnecting stream can otherwise
run for minutes). `should_stop` is polled while ffmpeg runs so a Stop KILLS the
subprocess at once — otherwise a downloader stuck in a long decode keeps the
agent "working" long after Stop. Empty on failure / stop / timeout."""
interval = max(0.5, float(interval_seconds or 4.0))
cap = max(1, int(max_frames or 64))
hdr = ["-headers", headers] if headers else []
with tempfile.TemporaryDirectory() as tmp:
pattern = os.path.join(tmp, "f_%05d.jpg")
cmd = ["ffmpeg", "-nostdin", "-loglevel", "error", *_RECONNECT, *hdr,
"-i", url, "-vf", f"fps=1/{interval}", "-frames:v", str(cap),
"-q:v", "3", pattern]
try:
subprocess.run(
["ffmpeg", "-nostdin", "-loglevel", "error", *_RECONNECT, *hdr,
"-i", url, "-vf", f"fps=1/{interval}", "-frames:v", str(cap),
"-q:v", "3", pattern],
check=True, timeout=timeout,
proc = subprocess.Popen(
cmd, stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)
except (subprocess.SubprocessError, FileNotFoundError):
except (OSError, ValueError):
return []
# Poll rather than block, so a Stop (or the per-video timeout) can kill a
# slow/wedged ffmpeg promptly instead of waiting it out.
start = time.monotonic()
while True:
try:
proc.wait(timeout=0.5)
break
except subprocess.TimeoutExpired:
if (should_stop and should_stop()) or (time.monotonic() - start > timeout):
_terminate(proc)
return []
if proc.returncode != 0:
return []
return _collect_frames(tmp, interval, cap)