bvandeusen/FabledCurator
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 35 Wiki Activity

fix(agent): server-side rate metrics + killable-on-stop ffmpeg
CI / lint (push) Successful in 2s
Details
CI / frontend-build (push) Successful in 19s
Details
CI / backend-lint-and-test (push) Successful in 25s
Details
CI / integration (push) Successful in 3m25s
Details

Browse Source 
Two follow-ups from live debugging of "work/min never populates" and
"stopped never reached".

1) jobs/min + downloads/min are now computed in the BACKEND on a fixed
   cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were
   derived client-side from poll deltas with a dt<30s guard — but a
   backgrounded/unfocused browser tab throttles its timers to ~1/min, so
   every delta exceeded 30s and the guard blanked the rates forever. A
   server-side rate is independent of how often the tab polls. Frontend just
   displays s.jobs_per_min / s.downloads_per_min. VERSION → .9.

2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a
   slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see
   the stop signal until ffmpeg returned, so Stop detached still-running
   threads and work kept flowing long after — "stopped" that wasn't really
   stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a
   `should_stop` callback every 0.5s, terminating (then killing) the process
   at once on Stop or the per-video timeout. A stop-killed job is handed back
   (transient), not failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-07-01 16:51:26 -04:00
parent 6282e753a9
commit 713a11e394
3 changed files with 91 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/fc_agent/app.py
+7 -23
View File
@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
# Bump on every agent change. The page embeds this and /status reports it; the UI
# warns to reload when they differ — so a stale browser-cached page can't be
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
VERSION = "2026-07-01.8 · real start/stop state machine (no more stuck 'stopping')"
VERSION = "2026-07-01.9 · server-computed jobs/min + downloads/min (poll-rate independent)"
logbuf.install()
cfg = Config.from_env()
@@ -252,11 +252,6 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
<script>
const PAGE_BUILD="__BUILD__"
let CAP=8
// Smoothed derived rates (jobs/min, downloads/min) + the deltas they're built
// from. The raw buffer/on-GPU/downloader counts change many times a second, so
// a poll only ever samples noise; these EWMA the two monotonic counters into a
// stable, readable throughput instead. Reset to null on load → re-converge.
let jpmv=null, dpmv=null, lastP=null, lastD=null, lastRateT=null
// Optimistic transitional state on click, then apply the POST's own status
// response (it returns worker.status()) for instant feedback — don't wait on the
// separate /status poll, which can lag behind the curator queue call.
@@ -308,23 +303,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
// backend truthfully lands on "stopped".
startbtn.disabled=(st!=='stopped')
stopbtn.disabled=!(running||st==='starting')
// Derived rates over the poll deltas — j/min ≈ GPU throughput, dl/min ≈ fetch
// throughput. EWMA-smoothed so a lull between jobs doesn't blank them; clamped
// at 0 so an agent restart (counters reset to 0 → negative delta) can't spike
// them; the dt<30s guard drops a backgrounded-tab gap that would read as huge.
const now=performance.now()
if(lastRateT!=null && s.processed!=null){
const dt=(now-lastRateT)/1000
if(dt>0.5 && dt<30){
const jp=Math.max(0,60*(s.processed-lastP)/dt)
const dp=Math.max(0,60*((s.downloaded||0)-lastD)/dt)
jpmv=(jpmv==null)?jp:0.4*jp+0.6*jpmv
dpmv=(dpmv==null)?dp:0.4*dp+0.6*dpmv
}
}
lastP=s.processed; lastD=(s.downloaded||0); lastRateT=now
jpm.textContent=(jpmv==null)?'—':Math.round(jpmv)
dpm.textContent=(dpmv==null)?'—':Math.round(dpmv)
// Throughput rates arrive READY from the backend (jobs/min ≈ GPU throughput,
// dl/min ≈ fetch throughput), computed there on a fixed cadence — so they show
// a real number no matter how often this tab polls (a backgrounded tab throttles
// its timers, which used to leave a client-side delta-rate blank forever).
jpm.textContent=(s.jobs_per_min!=null)?Math.round(s.jobs_per_min):'—'
dpm.textContent=(s.downloads_per_min!=null)?Math.round(s.downloads_per_min):'—'
done.textContent=s.processed
err.textContent=s.errors; err.className='n'+(s.errors>0?' warn':'')
waited.textContent=s.transient||0