bvandeusen
f0f031782d
Operator: the status tiles (state/active/processed) and the Start/Stop buttons freeze while the GPU meters stay live. Root cause: /status made an INLINE blocking curator call (queue_status) on every poll, and with curator buried under a 112k-job backlog that call stalled — freezing the whole status refresh (the GPU bars survived because /gpu is a lock-free local read). Made worse by the old util-band autoscaler, which grew workers toward the 32 cap forever because util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold — piling load onto curator and the agent process. - /status is now a pure in-memory read: worker.status() is lock-free, and the curator queue snapshot is refreshed by a background poller (never inline). - Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES: samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every ~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a grow stops helping it backs off one and holds, re-probing occasionally. No runaway, no flopping. - GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth, exposed on /gpu) when running, else smoothed client-side — so it glides instead of bouncing 0↔99. - act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the now-always-fast /status refresh recovers state regardless. - Log pane: bound the page to the viewport (height:100vh) so the Logs card scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at 400 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
45 lines
1.5 KiB
Python
45 lines
1.5 KiB
Python
"""In-memory log ring buffer so the control UI can show recent agent logs
|
|
(detector loads, job errors, autoscaler decisions, outage back-offs) without
|
|
needing `docker logs`. A bounded deque holds the last N formatted lines; a
|
|
logging.Handler appends to it; the UI polls /logs."""
|
|
import logging
|
|
from collections import deque
|
|
|
|
LINES: deque[str] = deque(maxlen=400)
|
|
|
|
|
|
class RingHandler(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
try:
|
|
LINES.append(self.format(record))
|
|
except Exception:
|
|
pass
|
|
|
|
|
|
_installed = False
|
|
|
|
|
|
def install(level: int = logging.INFO) -> None:
|
|
"""Attach the ring handler to the root logger once. fc_agent module loggers
|
|
propagate to root, so their records land here."""
|
|
global _installed
|
|
if _installed:
|
|
return
|
|
_installed = True
|
|
h = RingHandler()
|
|
h.setFormatter(
|
|
logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s", "%H:%M:%S")
|
|
)
|
|
root = logging.getLogger()
|
|
root.addHandler(h)
|
|
if root.level == logging.NOTSET or root.level > level:
|
|
root.setLevel(level)
|
|
# Keep the buffer signal-rich: silence the chatty HTTP/download libs (every
|
|
# HF model fetch logs per-request) so the console shows agent activity —
|
|
# detector loads, job errors, autoscale moves — not request spam.
|
|
for noisy in (
|
|
"uvicorn.access", "ultralytics", "httpx", "httpcore",
|
|
"huggingface_hub", "urllib3", "filelock",
|
|
):
|
|
logging.getLogger(noisy).setLevel(logging.WARNING)
|