bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 33 Wiki Activity

feat(agent): survive + auto-recover when curator is unreachable

Browse Source 
For redeploying curator while away with nobody to restart the agent:

- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
  401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
  fault. On a transport error it hands the job back (best effort) and signals
  the loop to back off — instead of calling fail(), which would burn the job's
  server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
  across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
  re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
  resets on the first successful lease, so a long outage is gentle and recovery
  is automatic within ≤60s of curator returning. Sleeps are interruptible so
  Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
  so a host reboot / crash-restart (restart: unless-stopped) self-heals with
  nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
  work" banner so the recovering state reads as recovery, not failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
Bryan Van Deusen
2026-06-30 08:33:33 -04:00
parent c6f38b0dac
commit 55fa4656ff
4 changed files with 105 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/fc_agent/app.py
+18 -1
View File
@@ -16,6 +16,16 @@ worker = Worker(cfg)
app = FastAPI(title="FabledCurator GPU agent")
@app.on_event("startup")
def _maybe_autostart() -> None:
# With AUTO_START set, a container restart (host reboot, or `restart:
# unless-stopped` after a crash) resumes the worker on its own — the slots
# then ride out a still-down curator via lease backoff. Lets the agent
# survive a redeploy with nobody at the desktop to click Start.
if cfg.auto_start and cfg.token:
worker.start()
@app.get("/", response_class=HTMLResponse)
def index() -> str:
return _PAGE
@@ -86,6 +96,10 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
<span class=stat><span class=n id=active>0</span><br>active now</span>
<span class=stat><span class=n id=done>0</span><br>processed</span>
<span class=stat><span class=n id=err>0</span><br>errors</span>
<span class=stat><span class=n id=wait>0</span><br>waited out</span>
</div>
<div id=banner style="display:none;margin:.6rem 0;padding:.5rem .8rem;border-radius:6px;background:#5a4a17;color:#ffe28a">
curator unreachable — holding work + retrying, will resume on its own (no restart needed)
</div>
<div class=gpu id=gpu>GPU — …</div>
<div class=bar><i id=gpubar style=width:0%></i></div>
@@ -103,7 +117,10 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
const s=await (await fetch('/status')).json()
CAP=s.max_concurrency||8; capn.textContent=CAP
state.textContent=s.state; active.textContent=s.active; done.textContent=s.processed
err.textContent=s.errors; fc.textContent=s.fc_url
err.textContent=s.errors; fc.textContent=s.fc_url; wait.textContent=s.transient||0
// Running but the queue read failed → curator is unreachable; show we're
// riding it out rather than erroring.
banner.style.display=(s.state==='running' && !s.queue)?'block':'none'
if(document.activeElement!==conc) conc.value=s.concurrency
conc.max=CAP
cfg.textContent=s.configured?'set':'MISSING'