fix(agent): real start/stop state machine — kill the stuck "stopping" pill

The Status pill hung on "stopping" forever (operator-flagged 2026-07-01). Root cause: the backend had no lifecycle state — status() only returned running/stopped — so the UI FABRICATED "stopping" in JS as `!running && active>0`. That pill only cleared when the backend's `active` counter hit 0, but stop() (a) blocked the HTTP handler on lease-release calls to curator and (b) left `active>0` whenever a consumer wedged mid-submit/release to an overloaded curator → "stopping" that never resolved. Give the backend a real, truthful state it drives itself: stopped → starting → running → stopping → stopped - start(): → starting; a downloader flips it to running on its FIRST successful lease (so "running" means curator is actually answering, not just "Start was clicked"). If curator's down it honestly stays "starting". - stop(): → stopping; returns immediately (no handler block). A background monitor waits for the worker threads to actually exit, releases leases, then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can NEVER hold the UI in "stopping" again. In-flight work is handed back safely. - Buttons follow the real state (Start only from stopped; both disabled through the transition), so you can't fight a transition. - Log every Start/Stop button press (routes) and every transition (worker), so the Logs panel shows exactly what each button did. Frontend now trusts s.state (drops the active>0 hack); VERSION → .8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 16:20:12 -04:00
parent 91ea06be79
commit 6282e753a9
2 changed files with 117 additions and 25 deletions
@@ -6,6 +6,8 @@ bandwidth for throughput), and watch GPU load + buffer occupancy + progress +
 the server-side queue. Config is env-seeded; the downloader count is adjustable
 here on the fly (GPU consumers autoscale between 1 and 2 on their own).
 """
+import logging
+
 from fastapi import FastAPI, Request
 from fastapi.responses import HTMLResponse, JSONResponse

@@ -14,10 +16,12 @@ from .config import Config
 from .gpu import read_gpu
 from .worker import Worker

+log = logging.getLogger("fc_agent.app")
+
 # Bump on every agent change. The page embeds this and /status reports it; the UI
 # warns to reload when they differ — so a stale browser-cached page can't be
 # mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
-VERSION = "2026-07-01.7 · status: smoothed jobs/min + downloads/min rates"
+VERSION = "2026-07-01.8 · real start/stop state machine (no more stuck 'stopping')"

 logbuf.install()
 cfg = Config.from_env()
@@ -53,12 +57,14 @@ def index() -> str:

@app.post("/start")
 def start():
+    log.info("UI: Start button pressed")   # the press; worker logs the transition
    worker.start()
    return JSONResponse(worker.status())


@app.post("/stop")
 def stop():
+    log.info("UI: Stop button pressed")
    worker.stop()
    return JSONResponse(worker.status())

@@ -260,10 +266,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  // /status refresh (now always fast) recovers the true state either way.
  const ac=new AbortController(); const to=setTimeout(()=>ac.abort(),8000)
  try{ applyStatus(await (await fetch('/'+p,{method:'POST',signal:ac.signal})).json()) }
-  catch{ /* leave the periodic refresh to recover the real state */ }
-  finally{ clearTimeout(to); startbtn.disabled=false; stopbtn.disabled=false }
+  catch{ refresh() /* on abort/error, repaint the real state from /status */ }
+  finally{ clearTimeout(to) }
 }
 function pending(label){
+  // Instant optimistic feedback on click; applyStatus (POST response, then the
+  // periodic poll) then owns the real state + which buttons are enabled.
  state.textContent=label; state.className='n busy'
  dot.className='dot amber'
  startbtn.disabled=true; stopbtn.disabled=true
@@ -284,15 +292,22 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
 }
 function applyStatus(s){
  CAP=s.max_concurrency||8; capn.textContent=CAP
-  const running=s.state==='running'
+  // The backend owns the state now (stopped|starting|running|stopping) and drives
+  // every transition, so the pill is always truthful — no client-side guessing
+  // from active>0, which used to wedge on "stopping" forever.
+  const st=s.state||'stopped'
+  const running=st==='running'
+  const busy=(st==='starting'||st==='stopping')
  // Stale-page guard: if the server is a newer build than this page, the cached
  // controls may misbehave — tell the operator to reload.
  if(s.build && s.build!==PAGE_BUILD) verbanner.style.display='block'
-  // "stopping" = stopped but in-flight jobs are still draining; resolves to
-  // "stopped" on its own once active hits 0.
-  const draining=!running && s.active>0
-  state.textContent=draining?'stopping':s.state
-  state.className='n'+(draining?' busy':'')
+  state.textContent=st
+  state.className='n'+(busy?' busy':'')
+  // Buttons follow the real state so you can't fight a transition: Start only
+  // from stopped; Stop only while up; both disabled through "stopping" until the
+  // backend truthfully lands on "stopped".
+  startbtn.disabled=(st!=='stopped')
+  stopbtn.disabled=!(running||st==='starting')
  // Derived rates over the poll deltas — j/min ≈ GPU throughput, dl/min ≈ fetch
  // throughput. EWMA-smoothed so a lull between jobs doesn't blank them; clamped
  // at 0 so an agent restart (counters reset to 0 → negative delta) can't spike
@@ -331,9 +346,16 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  if('configured' in s){
   const ok=s.configured
   fc.textContent=s.fc_url; cfg.textContent=ok?'set':'MISSING'
-   dot.className='dot '+(!ok?'red':(running?(s.queue?'green':'amber'):'amber'))
-   connlbl.textContent=!ok?'no token':(running?(s.queue?'running':'running · curator unreachable'):'stopped')
-   banner.style.display=(running && !s.queue)?'block':'none'
+   // Pill colour + label track the real state: green only when running AND
+   // curator is answering; amber for the transient states + a running-but-
+   // unreachable curator; grey when stopped; red with no token.
+   let dc='dot', lbl='stopped'
+   if(!ok){ dc='dot red'; lbl='no token' }
+   else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable' }
+   else if(st==='starting'){ dc='dot amber'; lbl='starting…' }
+   else if(st==='stopping'){ dc='dot amber'; lbl='stopping…' }
+   dot.className=dc; connlbl.textContent=lbl
+   banner.style.display=(st==='running' && !s.queue)?'block':'none'
   queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable'
  }
 }