Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10,
CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04:
- Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10.
- CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the
default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13:
ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open
"Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The
GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call,
not a hardware limit.
- PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed
(PEP 668); a single-purpose container owns its environment, so global installs
are fine and simplest.
- agent/ruff.toml pinned to py312 (was py310) so CI lints against the real
runtime; from __future__ import annotations stays (PEP 649 lazy annotations
are 3.14, so self-refs still evaluate on 3.12).
CI builds the image but has no GPU — validate on the desktop after pull that it
starts and loads CUDAExecutionProvider (not CPU fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent container (CUDA base, Python 3.10) crashed on startup with
`NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the
`from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14
(PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10.
CI lint/compile run on 3.14, so it slipped through.
- config.py: `from __future__ import annotations` so the self-referential
annotation is a string, never evaluated — works on 3.10 and every version.
- agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime)
and inherit the root rules. Ruff now flags exactly this class as F821, so CI's
lint lane catches it instead of shipping a broken image. (CI otherwise lints on
3.14, masking 3.10 issues.)
- client.py: submit path now retries in-place. A dedicated session with a
urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a
momentary blip after the GPU work is done doesn't discard it and force a full
re-download + recompute elsewhere. A duplicate submit after a lost response is
a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Addresses operator reports: Stop never finishes, the agent polls curator
constantly, and stale-cached pages get mistaken for a failed deploy.
- Stop is prompt: flip _running BEFORE any lock so /status + worker loops see
"stopped" immediately, and add a stop/shrink checkpoint in _process (after
decode, before the expensive detect+embed) that releases the job and bails —
so a Stop doesn't wait out heavy GPU work.
- Lazy curator polling: the queue snapshot is fetched only while a browser is
actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence,
not a constant background loop. The work loop's own lease/submit is curator's
only visitor otherwise — nothing polls just to poll.
- Build marker: VERSION is embedded in the page and reported on /status; the UI
shows a "reload" banner when they differ, so a browser-cached page can't be
mistaken for "the new image didn't deploy" (complements the no-store header).
CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the
GPU agent is linted + syntax-checked before its image builds (build.yml only
`docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator: the status tiles (state/active/processed) and the Start/Stop buttons
freeze while the GPU meters stay live. Root cause: /status made an INLINE
blocking curator call (queue_status) on every poll, and with curator buried
under a 112k-job backlog that call stalled — freezing the whole status refresh
(the GPU bars survived because /gpu is a lock-free local read). Made worse by the
old util-band autoscaler, which grew workers toward the 32 cap forever because
util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold —
piling load onto curator and the agent process.
- /status is now a pure in-memory read: worker.status() is lock-free, and the
curator queue snapshot is refreshed by a background poller (never inline).
- Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES:
samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every
~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a
grow stops helping it backs off one and holds, re-probing occasionally. No
runaway, no flopping.
- GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth,
exposed on /gpu) when running, else smoothed client-side — so it glides
instead of bouncing 0↔99.
- act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the
now-always-fast /status refresh recovers state regardless.
- Log pane: bound the page to the viewport (height:100vh) so the Logs card
scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at
400 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The control page is a static string served with no cache headers, so after
pulling a fresh agent image the browser kept showing the OLD UI until a hard
refresh (operator-flagged). Add a no-store middleware covering the page and the
status/gpu/logs polls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator: the buttons fire but the status view doesn't reflect the change. Cause:
act() ignored the POST's own status response and waited on the separate /status
poll (which lags behind the curator queue call). Now:
- act() applies the POST's returned status immediately for instant feedback, and
shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled)
the moment it's clicked.
- A stop that still has in-flight jobs draining shows "stopping" until active
hits 0, then resolves to "stopped" on its own.
- applyStatus() guards the /status-only fields (connection pill + queue) so the
lean action response can't blank them — the Start/Stop path deliberately skips
the slow curator call to stay snappy.
Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a
time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll +
autoscaler + /status share a single subprocess instead of piling up in the
server thread pool (which was what made clicks feel dead under load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two operator-reported issues with the GPU agent:
1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb
probed +1, judged it over a too-short noisy throughput window, saw no clear
gain and reverted -1 — every tick. Replace it with a GPU-utilization-band
controller: HOLD while smoothed util sits in a healthy band, grow only on
clear spare capacity (util below the low mark + VRAM headroom), shrink under
saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced
(DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool.
Load stays consistent instead of probe/reverting.
2. GPU util/VRAM bars only updated on manual refresh. They rode the /status
poll, which blocks on the curator queue call (slow when curator is busy), so
the meters froze between refreshes. Give them a dedicated /gpu endpoint
(local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop
the curator queue-status timeout 15s -> 5s so /status itself stays snappy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator meant the LOG section should fill down the viewport (vertical), not the
whole page going full-width horizontally. Restore the centered column (820px),
make .wrap a full-height flex column, and let the Logs card flex to fill the
remaining height to the bottom (drop the fixed 230px log-pane cap).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- Page fills the viewport horizontally (drop the 780px cap).
- Copy button on the Logs card → copies the console (clipboard API on localhost,
textarea-execCommand fallback), with a brief "Copied" confirmation.
- Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/
ultralytics to WARNING so the console shows agent activity (detector loads,
job errors, autoscale moves) instead of per-request HF-download spam.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root
logger; GET /logs serves it; the control page polls it into a console pane —
so runs are monitorable without `docker logs`. worker now logs autoscale moves
(one line per change, with jobs/s + util + VRAM) and job failures (job + image
+ reason); detectors already log load/disable.
- Restyled the whole control page: a proper dark layout with a header + live
connection pill, cards (Control / Status / Logs), a styled Auto switch +
worker stepper, status tiles, separate GPU-util and VRAM meters, and the log
console. No longer feels like an afterthought; all the existing control hooks
are preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The new per-job workload (3 detectors + several SigLIP embeds) is far more
GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and
is hard to guess. Add an Auto mode (default ON) that finds it:
- _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the
target: grow while throughput keeps improving and VRAM stays under budget,
revert a step that doesn't help, back off under memory pressure (VRAM >= 90%),
then settle and periodically re-probe (the GPU/IO balance shifts over a run).
- A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the
control UI re-enables it. status() reports `auto`; the dial reflects the
auto-chosen count (read-only) while Auto is on.
- AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
For redeploying curator while away with nobody to restart the agent:
- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
fault. On a transport error it hands the job back (best effort) and signals
the loop to back off — instead of calling fail(), which would burn the job's
server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
resets on the first successful lease, so a long outage is gentle and recovery
is automatic within ≤60s of curator returning. Sleeps are interruptible so
Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
so a host reboot / crash-restart (restart: unless-stopped) self-heals with
nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
work" banner so the recovering state reads as recovery, not failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
At 8 workers the GPU sat at ~5% util / <5GB VRAM — the pipeline is I/O-bound
(downloading + decoding images over HTTP), so the GPU starves until many workers
overlap that I/O. Raise MAX_CONCURRENCY 8→32 and make the UI worker control a
number input (reaching 32 by ±1 was tedious); the cap is reported via /status so
the UI clamps to it. Also size the shared requests pool (pool_maxsize=64) — the
default 10 would have throttled 32 workers + spammed "connection pool is full".
Verified by running; watch GPU util/VRAM climb as you dial up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Control UI gains what the operator asked for:
- GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how
hard the card is working while you're at the desktop.
- Worker count is now a live − / + control (POST /concurrency), not just an env:
the worker is a pool of independent slots (shared model, so slots add concurrent
inference, not N× VRAM). Dial up for speed, down to free the card. Replaces
pause/resume with Start/Stop + the worker dial.
- Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back
via client.release() so they're re-picked immediately (pairs with the server
recovery sweep).
Not CI-tested (agent/ outside CI) — verified by running.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP
(lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres
stay private. New top-level agent/ (outside CI scope — verified by running it):
- fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/
stop (stop frees the card; unprocessed leases expire + re-queue).
- fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed.
The two API seams to verify against the installed dghs-imgutils (flagged).
- fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence →
per-frame instances (the bag).
- fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client.
- fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress +
queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements +
README (token → build → run --gpus all → Start; CPU-fallback path).
This completes the CCIP pipeline end to end: agent produces region CCIP vectors →
RegionService stores → matcher suggests characters → rail. Verified by running on
the desktop (not CI). README calls out the imgutils API + model-string checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
bvandeusen