fix(agent): server-side rate metrics + killable-on-stop ffmpeg
Two follow-ups from live debugging of "work/min never populates" and "stopped never reached". 1) jobs/min + downloads/min are now computed in the BACKEND on a fixed cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were derived client-side from poll deltas with a dt<30s guard — but a backgrounded/unfocused browser tab throttles its timers to ~1/min, so every delta exceeded 30s and the guard blanked the rates forever. A server-side rate is independent of how often the tab polls. Frontend just displays s.jobs_per_min / s.downloads_per_min. VERSION → .9. 2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see the stop signal until ffmpeg returned, so Stop detached still-running threads and work kept flowing long after — "stopped" that wasn't really stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a `should_stop` callback every 0.5s, terminating (then killing) the process at once on Stop or the per-video timeout. A stop-killed job is handed back (transient), not failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+39
-9
@@ -5,6 +5,7 @@ import io
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from PIL import Image, ImageFile
|
||||
|
||||
@@ -105,28 +106,57 @@ def _collect_frames(
|
||||
return out
|
||||
|
||||
|
||||
def _terminate(proc: subprocess.Popen) -> None:
|
||||
"""Stop an ffmpeg cleanly, then hard-kill if it ignores SIGTERM."""
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=2)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
try:
|
||||
proc.wait(timeout=2)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
|
||||
|
||||
def sample_frames_from_url(
|
||||
url: str, interval_seconds: float, max_frames: int,
|
||||
*, headers: str = "", timeout: float = 1200.0,
|
||||
*, headers: str = "", timeout: float = 1200.0, should_stop=None,
|
||||
) -> list[tuple[float, Image.Image]]:
|
||||
"""Sample frames by pointing ffmpeg STRAIGHT at the media URL — it Range-reads
|
||||
only the video index + up to max_frames worth of content, so the agent never
|
||||
downloads the whole file (VR/4K originals run 800MB+ and would buffer ~1GB in
|
||||
RAM and get cut off mid-download). Reconnect flags resume a dropped transfer;
|
||||
the generous timeout lets it finish however slow the link is — completion over
|
||||
speed, and environment-agnostic (any HTTP+Range source). Empty on failure."""
|
||||
the timeout is the per-video ceiling (a slow/reconnecting stream can otherwise
|
||||
run for minutes). `should_stop` is polled while ffmpeg runs so a Stop KILLS the
|
||||
subprocess at once — otherwise a downloader stuck in a long decode keeps the
|
||||
agent "working" long after Stop. Empty on failure / stop / timeout."""
|
||||
interval = max(0.5, float(interval_seconds or 4.0))
|
||||
cap = max(1, int(max_frames or 64))
|
||||
hdr = ["-headers", headers] if headers else []
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
pattern = os.path.join(tmp, "f_%05d.jpg")
|
||||
cmd = ["ffmpeg", "-nostdin", "-loglevel", "error", *_RECONNECT, *hdr,
|
||||
"-i", url, "-vf", f"fps=1/{interval}", "-frames:v", str(cap),
|
||||
"-q:v", "3", pattern]
|
||||
try:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-nostdin", "-loglevel", "error", *_RECONNECT, *hdr,
|
||||
"-i", url, "-vf", f"fps=1/{interval}", "-frames:v", str(cap),
|
||||
"-q:v", "3", pattern],
|
||||
check=True, timeout=timeout,
|
||||
proc = subprocess.Popen(
|
||||
cmd, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
)
|
||||
except (subprocess.SubprocessError, FileNotFoundError):
|
||||
except (OSError, ValueError):
|
||||
return []
|
||||
# Poll rather than block, so a Stop (or the per-video timeout) can kill a
|
||||
# slow/wedged ffmpeg promptly instead of waiting it out.
|
||||
start = time.monotonic()
|
||||
while True:
|
||||
try:
|
||||
proc.wait(timeout=0.5)
|
||||
break
|
||||
except subprocess.TimeoutExpired:
|
||||
if (should_stop and should_stop()) or (time.monotonic() - start > timeout):
|
||||
_terminate(proc)
|
||||
return []
|
||||
if proc.returncode != 0:
|
||||
return []
|
||||
return _collect_frames(tmp, interval, cap)
|
||||
|
||||
Reference in New Issue
Block a user