Three safety/robustness fixes from the operator's run logs:
- Cap figures per frame (MAX_FIGURES, default 8) like components/panels already
are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds
of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept
(image 81602 looped on 413). This is the acute fix.
- Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still
exceed the cap (long video), keep the highest-scoring and log the drop, so a
submit body can never blow past curator's limit.
- Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the
meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the
reset can't drive it negative.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent container (CUDA base, Python 3.10) crashed on startup with
`NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the
`from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14
(PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10.
CI lint/compile run on 3.14, so it slipped through.
- config.py: `from __future__ import annotations` so the self-referential
annotation is a string, never evaluated — works on 3.10 and every version.
- agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime)
and inherit the root rules. Ruff now flags exactly this class as F821, so CI's
lint lane catches it instead of shipping a broken image. (CI otherwise lints on
3.14, masking 3.10 issues.)
- client.py: submit path now retries in-place. A dedicated session with a
urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a
momentary blip after the GPU work is done doesn't discard it and force a full
re-download + recompute elsewhere. A duplicate submit after a lost response is
a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Addresses operator reports: Stop never finishes, the agent polls curator
constantly, and stale-cached pages get mistaken for a failed deploy.
- Stop is prompt: flip _running BEFORE any lock so /status + worker loops see
"stopped" immediately, and add a stop/shrink checkpoint in _process (after
decode, before the expensive detect+embed) that releases the job and bails —
so a Stop doesn't wait out heavy GPU work.
- Lazy curator polling: the queue snapshot is fetched only while a browser is
actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence,
not a constant background loop. The work loop's own lease/submit is curator's
only visitor otherwise — nothing polls just to poll.
- Build marker: VERSION is embedded in the page and reported on /status; the UI
shows a "reload" banner when they differ, so a browser-cached page can't be
mistaken for "the new image didn't deploy" (complements the no-store header).
CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the
GPU agent is linted + syntax-checked before its image builds (build.yml only
`docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The new per-job workload (3 detectors + several SigLIP embeds) is far more
GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and
is hard to guess. Add an Auto mode (default ON) that finds it:
- _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the
target: grow while throughput keeps improving and VRAM stays under budget,
revert a step that doesn't help, back off under memory pressure (VRAM >= 90%),
then settle and periodically re-probe (the GPU/IO balance shifts over a run).
- A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the
control UI re-enables it. status() reports `auto`; the dial reflects the
auto-chosen count (read-only) while Auto is on.
- AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Better region PROPOSERS feeding the existing crop→SigLIP→max-over-bag heads (no
change to the learned-tagging approach; no per-tag cost — propose once, embed
each region, all heads in one matmul).
- detectors.py: lazy ultralytics YOLO wrapper, each proposer independently
optional + guarded (a bad weight spec / inference error self-disables that one,
logged, never breaks the worker). Weights resolve from an ultralytics name |
http(s) URL | "hf_repo::file", cached under HF_HOME. NMS merge so a figure two
detectors both find collapses to one crop.
- worker: figure boxes = imgutils detect_person ∪ general COCO person (merged)
→ CCIP + concept (anime + Western/realistic coverage); booru_yolo anatomy
components (head/cat-head/anatomy/…) → concept crops; comic panels → kind=
'panel' concept crops. Capped per frame (MAX_COMPONENTS/MAX_PANELS).
- config + compose: PERSON_WEIGHTS (default yolo11n.pt, works OOB),
ANATOMY_WEIGHTS + PANEL_WEIGHTS (operator sets booru_yolo URL + mosesb panel
hf::file; empty = off). ultralytics added to requirements.
- backend: image_region 'kind' doc notes 'panel'; no migration (free String,
and the bag scorer keys on a non-null siglip_embedding, not the kind, so any
SigLIP region joins the bag automatically).
Agent is outside CI — py-compiled here; operator tests on the GPU and checks
Western-vs-anime crop quality via /api/ccip observability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
For redeploying curator while away with nobody to restart the agent:
- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
fault. On a transport error it hands the job back (best effort) and signals
the loop to back off — instead of calling fail(), which would burn the job's
server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
resets on the first successful lease, so a long outage is gentle and recovery
is automatic within ≤60s of curator returning. Sleeps are interruptible so
Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
so a host reboot / crash-restart (restart: unless-stopped) self-heals with
nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
work" banner so the recovering state reads as recovery, not failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray,
lactation) that the whole-image SigLIP vector washes out: the GPU agent now
embeds figure crops with SigLIP too, stored as kind='concept' regions, and the
suggestion rail scores each image as a BAG (whole-image + every concept crop),
taking each head's MAX over the bag. The whole-image vector is always in the
bag, so this can never score lower than before.
Model-agnostic by construction: the server ANNOUNCES the embedding model
(HF name + version) in the lease, so the agent loads whatever the heads were
trained in and stays in lock-step — a model swap is a server setting + a
re-embed migration, never an agent change.
- agent: model-agnostic CropEmbedder (torch/transformers get_image_features,
fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits
figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the
back-catalogue backfill never churns figure/CCIP regions; torch cu124 +
transformers in the image.
- server: lease announces embed_model_name/embed_version; score_image is
max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill
'siglip' gates on a missing concept region (drains the back-catalogue,
retries failures, no double-enqueue); daily siglip-backfill beat; UI button;
/api/ccip/overview reports images_with_concept_siglip.
- v1 scope: suggestion rail only — auto-apply stays whole-image (conservative;
heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Control UI gains what the operator asked for:
- GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how
hard the card is working while you're at the desktop.
- Worker count is now a live − / + control (POST /concurrency), not just an env:
the worker is a pool of independent slots (shared model, so slots add concurrent
inference, not N× VRAM). Dial up for speed, down to free the card. Replaces
pause/resume with Start/Stop + the worker dial.
- Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back
via client.release() so they're re-picked immediately (pairs with the server
recovery sweep).
Not CI-tested (agent/ outside CI) — verified by running.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP
(lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres
stay private. New top-level agent/ (outside CI scope — verified by running it):
- fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/
stop (stop frees the card; unprocessed leases expire + re-queue).
- fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed.
The two API seams to verify against the installed dghs-imgutils (flagged).
- fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence →
per-frame instances (the bag).
- fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client.
- fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress +
queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements +
README (token → build → run --gpus all → Start; CPU-fallback path).
This completes the CCIP pipeline end to end: agent produces region CCIP vectors →
RegionService stores → matcher suggests characters → rail. Verified by running on
the desktop (not CI). README calls out the imgutils API + model-string checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
bvandeusen