Operator: the status tiles (state/active/processed) and the Start/Stop buttons
freeze while the GPU meters stay live. Root cause: /status made an INLINE
blocking curator call (queue_status) on every poll, and with curator buried
under a 112k-job backlog that call stalled — freezing the whole status refresh
(the GPU bars survived because /gpu is a lock-free local read). Made worse by the
old util-band autoscaler, which grew workers toward the 32 cap forever because
util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold —
piling load onto curator and the agent process.
- /status is now a pure in-memory read: worker.status() is lock-free, and the
curator queue snapshot is refreshed by a background poller (never inline).
- Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES:
samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every
~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a
grow stops helping it backs off one and holds, re-probing occasionally. No
runaway, no flopping.
- GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth,
exposed on /gpu) when running, else smoothed client-side — so it glides
instead of bouncing 0↔99.
- act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the
now-always-fast /status refresh recovers state regardless.
- Log pane: bound the page to the viewport (height:100vh) so the Logs card
scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at
400 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The control page is a static string served with no cache headers, so after
pulling a fresh agent image the browser kept showing the OLD UI until a hard
refresh (operator-flagged). Add a no-store middleware covering the page and the
status/gpu/logs polls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator: the buttons fire but the status view doesn't reflect the change. Cause:
act() ignored the POST's own status response and waited on the separate /status
poll (which lags behind the curator queue call). Now:
- act() applies the POST's returned status immediately for instant feedback, and
shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled)
the moment it's clicked.
- A stop that still has in-flight jobs draining shows "stopping" until active
hits 0, then resolves to "stopped" on its own.
- applyStatus() guards the /status-only fields (connection pill + queue) so the
lean action response can't blank them — the Start/Stop path deliberately skips
the slow curator call to stay snappy.
Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a
time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll +
autoscaler + /status share a single subprocess instead of piling up in the
server thread pool (which was what made clicks feel dead under load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two operator-reported issues with the GPU agent:
1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb
probed +1, judged it over a too-short noisy throughput window, saw no clear
gain and reverted -1 — every tick. Replace it with a GPU-utilization-band
controller: HOLD while smoothed util sits in a healthy band, grow only on
clear spare capacity (util below the low mark + VRAM headroom), shrink under
saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced
(DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool.
Load stays consistent instead of probe/reverting.
2. GPU util/VRAM bars only updated on manual refresh. They rode the /status
poll, which blocks on the curator queue call (slow when curator is busy), so
the meters froze between refreshes. Give them a dedicated /gpu endpoint
(local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop
the curator queue-status timeout 15s -> 5s so /status itself stays snappy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator meant the LOG section should fill down the viewport (vertical), not the
whole page going full-width horizontally. Restore the centered column (820px),
make .wrap a full-height flex column, and let the Logs card flex to fill the
remaining height to the bottom (drop the fixed 230px log-pane cap).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s,
truncated-image failures):
- BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo
components + panels) and embed them in ONE forward pass instead of one
forward+lock per crop. The per-crop path serialised every crop through the
inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating);
batching gives a real GPU-bound workload + far higher throughput. CCIP still
runs per figure inline.
- LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly-
truncated scraped images now load instead of failing the job 3× then erroring
("image file is truncated (N bytes not processed)").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- Page fills the viewport horizontally (drop the 780px cap).
- Copy button on the Logs card → copies the console (clipboard API on localhost,
textarea-execCommand fallback), with a brief "Copied" confirmation.
- Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/
ultralytics to WARNING so the console shows agent activity (detector loads,
job errors, autoscale moves) instead of per-request HF-download spam.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root
logger; GET /logs serves it; the control page polls it into a console pane —
so runs are monitorable without `docker logs`. worker now logs autoscale moves
(one line per change, with jobs/s + util + VRAM) and job failures (job + image
+ reason); detectors already log load/disable.
- Restyled the whole control page: a proper dark layout with a header + live
connection pill, cards (Control / Status / Logs), a styled Auto switch +
worker stepper, status tiles, separate GPU-util and VRAM meters, and the log
console. No longer feels like an afterthought; all the existing control hooks
are preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The new per-job workload (3 detectors + several SigLIP embeds) is far more
GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and
is hard to guess. Add an Auto mode (default ON) that finds it:
- _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the
target: grow while throughput keeps improving and VRAM stays under budget,
revert a step that doesn't help, back off under memory pressure (VRAM >= 90%),
then settle and periodically re-probe (the GPU/IO balance shifts over a run).
- A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the
control UI re-enables it. status() reports `auto`; the dial reflects the
auto-chosen count (read-only) while Auto is on.
- AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Better region PROPOSERS feeding the existing crop→SigLIP→max-over-bag heads (no
change to the learned-tagging approach; no per-tag cost — propose once, embed
each region, all heads in one matmul).
- detectors.py: lazy ultralytics YOLO wrapper, each proposer independently
optional + guarded (a bad weight spec / inference error self-disables that one,
logged, never breaks the worker). Weights resolve from an ultralytics name |
http(s) URL | "hf_repo::file", cached under HF_HOME. NMS merge so a figure two
detectors both find collapses to one crop.
- worker: figure boxes = imgutils detect_person ∪ general COCO person (merged)
→ CCIP + concept (anime + Western/realistic coverage); booru_yolo anatomy
components (head/cat-head/anatomy/…) → concept crops; comic panels → kind=
'panel' concept crops. Capped per frame (MAX_COMPONENTS/MAX_PANELS).
- config + compose: PERSON_WEIGHTS (default yolo11n.pt, works OOB),
ANATOMY_WEIGHTS + PANEL_WEIGHTS (operator sets booru_yolo URL + mosesb panel
hf::file; empty = off). ultralytics added to requirements.
- backend: image_region 'kind' doc notes 'panel'; no migration (free String,
and the bag scorer keys on a non-null siglip_embedding, not the kind, so any
SigLIP region joins the bag automatically).
Agent is outside CI — py-compiled here; operator tests on the GPU and checks
Western-vs-anime crop quality via /api/ccip observability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
For redeploying curator while away with nobody to restart the agent:
- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
fault. On a transport error it hands the job back (best effort) and signals
the loop to back off — instead of calling fail(), which would burn the job's
server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
resets on the first successful lease, so a long outage is gentle and recovery
is automatic within ≤60s of curator returning. Sleeps are interruptible so
Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
so a host reboot / crash-restart (restart: unless-stopped) self-heals with
nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
work" banner so the recovering state reads as recovery, not failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray,
lactation) that the whole-image SigLIP vector washes out: the GPU agent now
embeds figure crops with SigLIP too, stored as kind='concept' regions, and the
suggestion rail scores each image as a BAG (whole-image + every concept crop),
taking each head's MAX over the bag. The whole-image vector is always in the
bag, so this can never score lower than before.
Model-agnostic by construction: the server ANNOUNCES the embedding model
(HF name + version) in the lease, so the agent loads whatever the heads were
trained in and stays in lock-step — a model swap is a server setting + a
re-embed migration, never an agent change.
- agent: model-agnostic CropEmbedder (torch/transformers get_image_features,
fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits
figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the
back-catalogue backfill never churns figure/CCIP regions; torch cu124 +
transformers in the image.
- server: lease announces embed_model_name/embed_version; score_image is
max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill
'siglip' gates on a missing concept region (drains the back-catalogue,
retries failures, no double-enqueue); daily siglip-backfill beat; UI button;
/api/ccip/overview reports images_with_concept_siglip.
- v1 scope: suggestion rail only — auto-apply stays whole-image (conservative;
heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
At 8 workers the GPU sat at ~5% util / <5GB VRAM — the pipeline is I/O-bound
(downloading + decoding images over HTTP), so the GPU starves until many workers
overlap that I/O. Raise MAX_CONCURRENCY 8→32 and make the UI worker control a
number input (reaching 32 by ±1 was tedious); the cap is reported via /status so
the UI clamps to it. Also size the shared requests pool (pool_maxsize=64) — the
default 10 would have throttled 32 workers + spammed "connection pool is full".
Verified by running; watch GPU util/VRAM climb as you dial up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
A naive convert('RGB') on a palette-with-transparency image (common: character
PNGs on a clear background) lets PIL guess the transparent pixels — black-ish
artifacts that bleed into the crop + the CCIP embedding (and the "should be
converted to RGBA" warning). to_rgb() composites over white first for a clean,
consistent background; used by both stills and video frames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Control UI gains what the operator asked for:
- GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how
hard the card is working while you're at the desktop.
- Worker count is now a live − / + control (POST /concurrency), not just an env:
the worker is a pool of independent slots (shared model, so slots add concurrent
inference, not N× VRAM). Dial up for speed, down to free the card. Replaces
pause/resume with Start/Stop + the worker dial.
- Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back
via client.release() so they're re-picked immediately (pairs with the server
recovery sweep).
Not CI-tested (agent/ outside CI) — verified by running.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
onnxruntime-gpu needs cuDNN 9; the plain cuda:12.4.1-runtime image lacks it
(libcudnn.so.9 missing → CUDAExecutionProvider falls back to CPU). Switch to
the -cudnn-runtime variant which bundles cuDNN 9.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
compose file (pull the published image, GPU reservation, model-cache volume,
.env for the token) so the agent runs with `docker compose up -d` instead of a
long docker run. A copy + .env template also placed in ~/Documents/fc-gpu-agent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Build + push fabledcurator-agent alongside web/ml (own CUDA + onnxruntime-gpu
image, context=agent/, same tag cadence: main → :main/:latest/:c-<sha>, tag →
:<version>). So the operator PULLS + runs it on the GPU machine instead of
building locally. README switched to docker pull.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP
(lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres
stay private. New top-level agent/ (outside CI scope — verified by running it):
- fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/
stop (stop frees the card; unprocessed leases expire + re-queue).
- fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed.
The two API seams to verify against the installed dghs-imgutils (flagged).
- fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence →
per-frame instances (the bag).
- fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client.
- fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress +
queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements +
README (token → build → run --gpus all → Start; CPU-fallback path).
This completes the CCIP pipeline end to end: agent produces region CCIP vectors →
RegionService stores → matcher suggests characters → rail. Verified by running on
the desktop (not CI). README calls out the imgutils API + model-string checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
bvandeusen