bvandeusen
55fa4656ff
For redeploying curator while away with nobody to restart the agent: - _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx, 401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job fault. On a transport error it hands the job back (best effort) and signals the loop to back off — instead of calling fail(), which would burn the job's server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't re-lease forever. - lease loop retries with capped exponential backoff (poll_idle → 60s) and resets on the first successful lease, so a long outage is gentle and recovery is automatic within ≤60s of curator returning. Sleeps are interruptible so Stop / pool-shrink stays responsive. - AUTO_START env (default on in compose) resumes the worker on container start, so a host reboot / crash-restart (restart: unless-stopped) self-heals with nobody at the desktop. - control UI shows a "waited out" counter + an "curator unreachable, holding work" banner so the recovering state reads as recovery, not failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
135 lines
5.4 KiB
Python
135 lines
5.4 KiB
Python
"""FastAPI control surface for the agent (served on localhost).
|
||
|
||
Start / stop the worker pool, tune the worker count live (trades desktop
|
||
responsiveness for throughput), and watch GPU load + progress + the server-side
|
||
queue. Config is env-seeded; the worker count is adjustable here on the fly.
|
||
"""
|
||
from fastapi import FastAPI, Request
|
||
from fastapi.responses import HTMLResponse, JSONResponse
|
||
|
||
from .config import Config
|
||
from .gpu import read_gpu
|
||
from .worker import Worker
|
||
|
||
cfg = Config.from_env()
|
||
worker = Worker(cfg)
|
||
app = FastAPI(title="FabledCurator GPU agent")
|
||
|
||
|
||
@app.on_event("startup")
|
||
def _maybe_autostart() -> None:
|
||
# With AUTO_START set, a container restart (host reboot, or `restart:
|
||
# unless-stopped` after a crash) resumes the worker on its own — the slots
|
||
# then ride out a still-down curator via lease backoff. Lets the agent
|
||
# survive a redeploy with nobody at the desktop to click Start.
|
||
if cfg.auto_start and cfg.token:
|
||
worker.start()
|
||
|
||
|
||
@app.get("/", response_class=HTMLResponse)
|
||
def index() -> str:
|
||
return _PAGE
|
||
|
||
|
||
@app.post("/start")
|
||
def start():
|
||
worker.start()
|
||
return JSONResponse(worker.status())
|
||
|
||
|
||
@app.post("/stop")
|
||
def stop():
|
||
worker.stop()
|
||
return JSONResponse(worker.status())
|
||
|
||
|
||
@app.post("/concurrency")
|
||
async def concurrency(request: Request):
|
||
body = await request.json()
|
||
worker.set_concurrency(int(body.get("value", 1)))
|
||
return JSONResponse(worker.status())
|
||
|
||
|
||
@app.get("/status")
|
||
def status():
|
||
s = worker.status()
|
||
s["fc_url"] = cfg.fc_url
|
||
s["configured"] = bool(cfg.token)
|
||
s["gpu"] = read_gpu()
|
||
try:
|
||
s["queue"] = worker.client.queue_status()
|
||
except Exception:
|
||
s["queue"] = None
|
||
return JSONResponse(s)
|
||
|
||
|
||
_PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
||
<title>FabledCurator GPU agent</title>
|
||
<style>
|
||
body{font:14px system-ui;margin:2rem;max-width:680px;background:#14171a;color:#e8e8e8}
|
||
h1{font-size:18px} button{font:14px system-ui;padding:.5rem 1rem;border:0;border-radius:6px;
|
||
margin-right:.5rem;cursor:pointer;color:#fff} .start{background:#2e7d32}.stop{background:#b3261e}
|
||
.step{background:#33373b;padding:.4rem .7rem;font-weight:700}
|
||
.stat{display:inline-block;margin-right:1.5rem;vertical-align:top}
|
||
.n{font-size:22px;font-weight:700} code{background:#222;padding:2px 6px;border-radius:4px}
|
||
.q,.gpu{margin-top:1rem;color:#9aa} .bar{height:8px;border-radius:4px;background:#222;overflow:hidden;
|
||
max-width:320px;margin-top:4px} .bar>i{display:block;height:100%;background:#3f7d3f}
|
||
.row{margin:.8rem 0}
|
||
</style></head><body>
|
||
<h1>FabledCurator GPU agent</h1>
|
||
<p>FC: <code id=fc>—</code> · token <code id=cfg>—</code></p>
|
||
<div class=row>
|
||
<button class=start onclick=act('start')>Start</button>
|
||
<button class=stop onclick=act('stop')>Stop</button>
|
||
</div>
|
||
<div class=row>
|
||
workers
|
||
<button class=step onclick=setc(-1)>−</button>
|
||
<input id=conc type=number min=1 value=1
|
||
style="width:3.5rem;font:700 16px system-ui;text-align:center;background:#222;color:#e8e8e8;border:1px solid #444;border-radius:6px;padding:.3rem"
|
||
onchange="setv(this.value)">
|
||
<button class=step onclick=setc(1)>+</button>
|
||
<span class=cap style=color:#9aa>(more = overlap I/O, fill the GPU) max <b id=capn>8</b></span>
|
||
</div>
|
||
<div class=row>
|
||
<span class=stat><span class=n id=state>stopped</span><br>state</span>
|
||
<span class=stat><span class=n id=active>0</span><br>active now</span>
|
||
<span class=stat><span class=n id=done>0</span><br>processed</span>
|
||
<span class=stat><span class=n id=err>0</span><br>errors</span>
|
||
<span class=stat><span class=n id=wait>0</span><br>waited out</span>
|
||
</div>
|
||
<div id=banner style="display:none;margin:.6rem 0;padding:.5rem .8rem;border-radius:6px;background:#5a4a17;color:#ffe28a">
|
||
curator unreachable — holding work + retrying, will resume on its own (no restart needed)
|
||
</div>
|
||
<div class=gpu id=gpu>GPU — …</div>
|
||
<div class=bar><i id=gpubar style=width:0%></i></div>
|
||
<div class=q id=queue></div>
|
||
<script>
|
||
let CAP=8
|
||
async function act(p){await fetch('/'+p,{method:'POST'});refresh()}
|
||
function setc(d){ setv((parseInt(conc.value||'1'))+d) }
|
||
async function setv(v){
|
||
v=Math.max(1,Math.min(CAP,parseInt(v)||1)); conc.value=v
|
||
await fetch('/concurrency',{method:'POST',headers:{'Content-Type':'application/json'},
|
||
body:JSON.stringify({value:v})});refresh()
|
||
}
|
||
async function refresh(){
|
||
const s=await (await fetch('/status')).json()
|
||
CAP=s.max_concurrency||8; capn.textContent=CAP
|
||
state.textContent=s.state; active.textContent=s.active; done.textContent=s.processed
|
||
err.textContent=s.errors; fc.textContent=s.fc_url; wait.textContent=s.transient||0
|
||
// Running but the queue read failed → curator is unreachable; show we're
|
||
// riding it out rather than erroring.
|
||
banner.style.display=(s.state==='running' && !s.queue)?'block':'none'
|
||
if(document.activeElement!==conc) conc.value=s.concurrency
|
||
conc.max=CAP
|
||
cfg.textContent=s.configured?'set':'MISSING'
|
||
if(s.gpu){
|
||
gpu.textContent=`GPU — ${s.gpu.util_pct}% util · VRAM ${s.gpu.mem_used_mb}/${s.gpu.mem_total_mb} MB · ${s.gpu.temp_c}°C`
|
||
gpubar.style.width=Math.round(100*s.gpu.mem_used_mb/s.gpu.mem_total_mb)+'%'
|
||
} else { gpu.textContent='GPU — n/a (CPU fallback?)'; gpubar.style.width='0%' }
|
||
queue.textContent=s.queue?`queue — pending ${s.queue.pending} · in flight ${s.queue.leased} · done ${s.queue.done} · errored ${s.queue.error}`:'queue — unreachable'
|
||
}
|
||
refresh(); setInterval(refresh,3000)
|
||
</script></body></html>"""
|