feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms), so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit) left the fast GPU idle during every download. Rearchitect worker.py into a producer/consumer pipeline: downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU consumers (detect+embed→submit) - Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a 2nd consumer if it has util/VRAM headroom and lifts throughput, else trim). - Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping RAM + lease look-ahead. VRAM pressure sheds a consumer immediately. - Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU; curator's 180s TTL would otherwise reclaim them mid-buffer). - Preserves all resilience: lease exp-backoff, submit-path retry (#169), release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools and releases every held lease at once (single held-set as source of truth). - Consumers SHARE one embedder + proposers instance (a 2nd consumer adds concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots). - UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU · processed · errors, a buffer-occupancy meter, and a consumers/waited-out line; the dial now tunes downloaders. Build marker 2026-07-01.1. Also fix the operator-flagged detector warning: yolo11n + the comic-panel model threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load- time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3 crop proposers and spamming the log per image. Disable that fusion (unfused inference is correct, marginally slower) and permanently self-disable a proposer on the first inference failure instead of re-throwing forever. Refs milestone 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
+26
-13
@@ -1,8 +1,10 @@
|
||||
"""FastAPI control surface for the agent (served on localhost).
|
||||
|
||||
Start / stop the worker pool, tune the worker count live (trades desktop
|
||||
responsiveness for throughput), and watch GPU load + progress + the server-side
|
||||
queue. Config is env-seeded; the worker count is adjustable here on the fly.
|
||||
Start / stop the download→GPU pipeline, tune the downloader count live (the
|
||||
workload is download-bound, so downloaders are the dial that trades desktop
|
||||
bandwidth for throughput), and watch GPU load + buffer occupancy + progress +
|
||||
the server-side queue. Config is env-seeded; the downloader count is adjustable
|
||||
here on the fly (GPU consumers autoscale between 1 and 2 on their own).
|
||||
"""
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
@@ -15,7 +17,7 @@ from .worker import Worker
|
||||
# Bump on every agent change. The page embeds this and /status reports it; the UI
|
||||
# warns to reload when they differ — so a stale browser-cached page can't be
|
||||
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
|
||||
VERSION = "2026-06-30.10 · video region early-exit"
|
||||
VERSION = "2026-07-01.1 · download/GPU pipeline"
|
||||
|
||||
logbuf.install()
|
||||
cfg = Config.from_env()
|
||||
@@ -156,9 +158,9 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
||||
#conc{width:3.4rem;height:32px;text-align:center;font:700 16px system-ui;background:#11151a;
|
||||
color:var(--fg);border:1px solid var(--bd);border-radius:8px}
|
||||
.hint{color:var(--mut);font-size:12px;margin-top:12px}
|
||||
.tiles{display:grid;grid-template-columns:repeat(5,1fr);gap:10px;margin-bottom:16px}
|
||||
.tile{background:#13171d;border:1px solid var(--bd);border-radius:10px;padding:12px 10px;text-align:center}
|
||||
.tile .n{font:800 24px ui-monospace,monospace;line-height:1.1}
|
||||
.tiles{display:grid;grid-template-columns:repeat(6,1fr);gap:8px;margin-bottom:16px}
|
||||
.tile{background:#13171d;border:1px solid var(--bd);border-radius:10px;padding:12px 8px;text-align:center}
|
||||
.tile .n{font:800 22px ui-monospace,monospace;line-height:1.1}
|
||||
.tile .n.warn{color:var(--red)} .tile .n.ok{color:var(--grn)}
|
||||
.tile .l{font-size:10px;text-transform:uppercase;letter-spacing:.06em;color:var(--mut);margin-top:4px}
|
||||
.meters{display:flex;flex-direction:column;gap:10px;margin-bottom:14px}
|
||||
@@ -167,6 +169,7 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
||||
.bar{height:9px;border-radius:5px;background:#11151a;border:1px solid var(--bd);overflow:hidden}
|
||||
.bar>i{display:block;height:100%;width:0;background:linear-gradient(90deg,#3a7d57,var(--grn));transition:width .4s}
|
||||
#utilbar{background:linear-gradient(90deg,#9a5a1f,var(--acc))}
|
||||
#bufbar{background:linear-gradient(90deg,#2f5a9a,#4a86d8)}
|
||||
.queue{font:13px ui-monospace,monospace;color:var(--mut)}
|
||||
.banner{margin:0 0 14px;padding:.7rem .9rem;border-radius:10px;background:#3a2f12;
|
||||
border:1px solid #5a4a17;color:#ffd98a;font-size:13px}
|
||||
@@ -208,24 +211,28 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
||||
<button class=step onclick=setc(1)>+</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class=hint id=conchint>auto-tuning to fill the GPU · max <b id=capn>8</b></div>
|
||||
<div class=hint id=conchint>auto-tuning downloaders to keep the GPU fed · max <b id=capn>8</b></div>
|
||||
</section>
|
||||
|
||||
<section class=card>
|
||||
<div class=card-h>Status</div>
|
||||
<div class=tiles>
|
||||
<div class=tile><div class=n id=state>—</div><div class=l>state</div></div>
|
||||
<div class=tile><div class=n id=active>0</div><div class=l>active</div></div>
|
||||
<div class=tile><div class=n id=dln>0</div><div class=l>downloaders</div></div>
|
||||
<div class=tile><div class=n id=bufn>—</div><div class=l>buffer</div></div>
|
||||
<div class=tile><div class=n id=active>0</div><div class=l>on GPU</div></div>
|
||||
<div class=tile><div class="n ok" id=done>0</div><div class=l>processed</div></div>
|
||||
<div class=tile><div class=n id=err>0</div><div class=l>errors</div></div>
|
||||
<div class=tile><div class=n id=wait>0</div><div class=l>waited out</div></div>
|
||||
</div>
|
||||
<div class=meters>
|
||||
<div class=meter><div class=meter-h><span>GPU util</span><b id=utillbl>—</b></div>
|
||||
<div class=bar><i id=utilbar></i></div></div>
|
||||
<div class=meter><div class=meter-h><span>VRAM</span><b id=vramlbl>—</b></div>
|
||||
<div class=bar><i id=gpubar></i></div></div>
|
||||
<div class=meter><div class=meter-h><span>buffer occupancy</span><b id=buflbl>—</b></div>
|
||||
<div class=bar><i id=bufbar></i></div></div>
|
||||
</div>
|
||||
<div class=queue id=pipe>consumers — · waited out 0</div>
|
||||
<div class=queue id=queue>queue —</div>
|
||||
</section>
|
||||
|
||||
@@ -281,13 +288,19 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
||||
const draining=!running && s.active>0
|
||||
state.textContent=draining?'stopping':s.state
|
||||
state.className='n'+(draining?' busy':'')
|
||||
active.textContent=s.active; done.textContent=s.processed
|
||||
dln.textContent=(s.downloaders!=null?s.downloaders:'—')
|
||||
bufn.textContent=(s.buffer!=null?(s.buffer+'/'+s.buffer_max):'—')
|
||||
active.textContent=s.active; active.className='n'+(s.active>0?' busy':'')
|
||||
done.textContent=s.processed
|
||||
err.textContent=s.errors; err.className='n'+(s.errors>0?' warn':'')
|
||||
wait.textContent=s.transient||0
|
||||
pipe.textContent='consumers '+(s.consumers!=null?s.consumers:'—')+' · waited out '+(s.transient||0)
|
||||
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
|
||||
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
|
||||
buflbl.textContent=s.buffer+' / '+s.buffer_max; bufbar.style.width=p+'%' }
|
||||
// Auto on → dial reflects the auto-chosen count (read-only); off → manual.
|
||||
if(document.activeElement!==autochk) autochk.checked=!!s.auto
|
||||
conc.disabled=!!s.auto; conc.style.opacity=s.auto?0.55:1
|
||||
conchint.textContent=s.auto?('auto-tuning to fill the GPU · max '+CAP):('manual · max '+CAP)
|
||||
conchint.textContent=s.auto?('auto-tuning downloaders to keep the GPU fed · max '+CAP):('manual downloaders · max '+CAP)
|
||||
if(document.activeElement!==conc) conc.value=s.concurrency
|
||||
conc.max=CAP
|
||||
// Connection pill + queue come only from the /status poll (the Start/Stop POST
|
||||
|
||||
Reference in New Issue
Block a user