49 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	2677ce020c	fix: the ml artifact was in a second list I never grepped for (4311) ... ac70f2a removed the `ml` image but CI went red on six tests: `tests/test_artifact_identity.py` parametrises over its own `ARTIFACTS = ("web", "ml", "agent", "extension")`, and every case now hits the dispatch guard that same commit added. My miss, and a specific one. I grepped for `fabledcurator-ml` and `ML_PATHS` and called the survey done — but the artifact is also named as a bare `"ml"`, which neither pattern finds. Rule 90 (grep pinned tests when changing a shared symbol) was surfaced to me while I was making the change and I ran a narrower sweep than it asks for. Lesson #4275 names the shape exactly: an absence claim is only as good as the search behind it, and a grep that matched nothing looks identical to a grep that asked the wrong question. The re-run was done by value, not by name: every occurrence of a bare `ml` in the repo, then filtering. That distinguishes the two things the token means — the celery LANE `ml` and the `backend/app/services/ml` package both stay and account for nearly every hit; only the IMAGE name went. Worth stating in the test, since the next person to grep will hit the same ambiguity. Also swept the prose the first pass left describing the old pipeline: "Four artifacts" (README, ci-requirements), "leaves web and ml alone" (×5 in build.yml, plus both docs), "CI publishes it alongside the web/ml images" (agent/README). The run 4896 build-time measurements keep their `ml` number — that was measured when ml was a real build — with a note saying the name has since gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:32:49 -04:00
bvandeusenandClaude Opus 4.8	57e52433d0	feat(agent): idle-unload GPU models to free VRAM when the queue is idle ... The SigLIP embedder + YOLO proposers load lazily then stay resident for the container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded. Add a monitor thread that unloads the torch-owned models after cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then torch.cuda.empty_cache() to hand the blocks back to the driver. They reload lazily on the next job via the existing _ensure_embedder / _proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in /status (models_loaded) and the agent UI pipe line; the VRAM meter drops too. Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa	2026-07-17 12:57:31 -04:00
bvandeusenandClaude Opus 4.8	a4df279343	feat(ml): lease announces detector config; agent builds proposers from it live (#134 step 2) ... The GPU lease now carries the crop-proposer config from MLSettings in a per-job 'detectors' block (same pattern as embed_model_name). The agent's worker builds its Proposers from the announced config via _effective_cfg (lease block overlaid on env) + _proposers_for (rebuilds only when a config signature changes) — so an operator's UI edit takes effect on the next lease with NO restart, and env is now just the bootstrap fallback until the server announces. enabled-off maps to empty weights (proposer skipped); dedupe_iou + max_regions also come from the effective cfg. Test: lease announces the detectors block with the seeded default weights. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-05 19:42:59 -04:00
bvandeusenandClaude Fable 5	b1cfbcc06a	fix(agent): sleep mode — an empty queue sheds to one downloader and backs the lease poll off to 15 min ... Operator-flagged on the deployed .5 build: the autoscaler grew the pool 1→8 against an EMPTY queue (an empty buffer read as 'GPU starving' regardless of WHY), and every downloader kept polling lease every 10s all night. - New idle signal straight from the lease results: an empty lease sets _idle, any jobs clear it. The occupancy-low branch now distinguishes three cases: queue empty → shed to ONE polling downloader; pinned at the bandwidth cap → shed toward 3; cap headroom + work flowing → grow. - Idle lease polls back off exponentially per downloader to IDLE_POLL_MAX_SECONDS (15 min) and reset the moment work appears — so an idle night costs one HTTP call per 15 min, and new work is noticed within at most ~15 min (operator-accepted trade-off). - UI hint: 'idle — queue empty, lease poll backed off'; /status gains idle. Agent build 2026-07-02.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 20:32:11 -04:00
bvandeusenandClaude Fable 5	7c19ad91ed	feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback) ... Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint says 'holding at the bandwidth cap' and /status reports bw_capped, so the behavior is legible without tests that need the ML stack. Reset content tagging: stays a FULL-instance reset (operator's call), but now lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale counts are rejected server-side). Copy no longer claims suggestions repopulate: it says plainly the heads' training examples are deleted and re-tagging starts fresh. Moved out of TagMaintenanceCard into DangerZoneCard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:14:48 -04:00
bvandeusenandClaude Fable 5	95d2ae1d58	feat(agent): global bandwidth cap — the agent can't saturate the desktop's network ... One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited; live MB/s dial + net readout in the control UI) is charged by every still download (streamed chunk reads) and every ffmpeg video stream (metered from outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget). Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8 unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's browser included). Capping the aggregate keeps the desktop usable and still beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 11:20:45 -04:00
bvandeusenandClaude Fable 5	09e2772628	fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards ... The hourly ccip backfill's skip-list lacked 'error' (and the daily siglip/embed variants re-gated failures on their missing results), so every permanently-bad file got a fresh doomed job each run — ~24 duplicate error rows/day per file, the perpetual 'unprocessable' flood. An errored job is now a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via /retry_errors (an errored back-catalogue needs one button press after a model swap). One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements) runs before every backfill and inside /retry_errors: error rows made moot by a later pending/leased/done row go first, then older duplicates (newest reason survives) — so the error count reads as distinct failing files and a retry can't fan one file out into duplicate pending jobs. /retry_errors now returns {requeued, pruned} and the toast shows both. Poison-loop guards (release and lease-expiry burn no attempts, so a job that stalls its transfer or crashes the agent every time cycled forever — operator-observed jobs 99044/125288/131594/143131): - agent: 3 in-session transient bounces (fetch or submit) → fail with the real reason instead of another release; strikes never count while stopping, and clear on submit success. Agent build 2026-07-02.3. - server: the 60s orphan sweep (statements shared between the beat task and GpuJobService so they can't drift) converts expired leases with >=5 lease grants and pending jobs with >=10 to 'error', preserving the last stored failure reason. Backstops old agent builds. Tests: tombstone rule across all three backfill variants, moot-row pruning, poison conversions, and the extended /retry_errors dedupe contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-01 22:52:38 -04:00
bvandeusenandClaude Fable 5	1b1d3732dc	feat(agent): store ffmpeg's actual failure reason in the job's error field ... sample_frames_from_url now returns (frames, reason) — reason carries the SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found", or the timeout) instead of only logging it agent-side. The worker folds it into the failure it reports, so curator's GpuJob.error reads e.g. no frames sampled from video — ffmpeg exit 183: moov atom not found ... instead of the bare "(unprocessable)". The errored-jobs list becomes self-describing: after a retry sweep, surviving errors name their real defect without needing the agent log. Return-value plumbing (not shared state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-01 22:01:20 -04:00
bvandeusenandClaude Fable 5	3a683d7feb	fix(agent): short videos failed as "unprocessable" — fps filter emits 0 frames ... Root cause of the "no frames sampled from video (unprocessable)" flood (operator-flagged 2026-07-02, whole 62k-70k image block + others): the sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4) frames — which is ZERO for any clip shorter than ~2s. Short animation loops (0.5s, 1.75s — verified against two originals from different artists) are complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames, exited 0, and the agent failed the job as unprocessable. Long videos worked, so only the short-clip class flooded. Fix: sample with select ("first frame always, then one per interval of timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range yuv420p sources don't trip the mjpeg encoder's full-range strictness (secondary failure observed on a 4440x2760 clip). Verified locally against both failing originals (frames extracted, PIL-clean) and a synthetic 15s video (4 frames at t=0/4/8/12 — long-video behavior unchanged). Observability (why this hid for weeks): ffmpeg's stderr was discarded, so every failure logged only "no frames sampled". stderr now goes to a temp file and its tail is logged on any produced-no-frames/timeout failure — the log names the actual ffmpeg reason from now on. Also: frames written before a mid-stream ffmpeg error are now kept (partial > nothing). VERSION → 2026-07-02.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-01 21:00:29 -04:00
bvandeusenandClaude Opus 4.8	0a618db10c	fix(agent): Status froze after one update — conchint.textContent destroyed #capn ... THE root cause of "the Status section doesn't update" (chased across several rounds; the backend was always healthy). `#capn` (the max-concurrency number) was nested inside `#conchint`: <div id=conchint>… · max <b id=capn>8</b></div> and applyStatus() ran, every call: `capn.textContent=CAP` AND `conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces ALL of conchint's children — destroying the <b id=capn> node. So: call 1: capn exists → tiles update → conchint.textContent DELETES capn call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) → applyStatus throws on its FIRST line → aborts before any tile → frozen. This is exactly the observed "ticks a couple times then freezes", and why /gpu + /logs (which never touch capn) kept updating fine. The capn write was redundant anyway — conchint.textContent already renders the max. Remove the nested <b id=capn> element and the capn.textContent line; the hint still shows "· max N". VERSION → .10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 20:19:37 -04:00
bvandeusenandClaude Opus 4.8	713a11e394	fix(agent): server-side rate metrics + killable-on-stop ffmpeg ... Two follow-ups from live debugging of "work/min never populates" and "stopped never reached". 1) jobs/min + downloads/min are now computed in the BACKEND on a fixed cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were derived client-side from poll deltas with a dt<30s guard — but a backgrounded/unfocused browser tab throttles its timers to ~1/min, so every delta exceeded 30s and the guard blanked the rates forever. A server-side rate is independent of how often the tab polls. Frontend just displays s.jobs_per_min / s.downloads_per_min. VERSION → .9. 2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see the stop signal until ffmpeg returned, so Stop detached still-running threads and work kept flowing long after — "stopped" that wasn't really stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a `should_stop` callback every 0.5s, terminating (then killing) the process at once on Stop or the per-video timeout. A stop-killed job is handed back (transient), not failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:51:26 -04:00
bvandeusenandClaude Opus 4.8	6282e753a9	fix(agent): real start/stop state machine — kill the stuck "stopping" pill ... The Status pill hung on "stopping" forever (operator-flagged 2026-07-01). Root cause: the backend had no lifecycle state — status() only returned running/stopped — so the UI FABRICATED "stopping" in JS as `!running && active>0`. That pill only cleared when the backend's `active` counter hit 0, but stop() (a) blocked the HTTP handler on lease-release calls to curator and (b) left `active>0` whenever a consumer wedged mid-submit/release to an overloaded curator → "stopping" that never resolved. Give the backend a real, truthful state it drives itself: stopped → starting → running → stopping → stopped - start(): → starting; a downloader flips it to running on its FIRST successful lease (so "running" means curator is actually answering, not just "Start was clicked"). If curator's down it honestly stays "starting". - stop(): → stopping; returns immediately (no handler block). A background monitor waits for the worker threads to actually exit, releases leases, then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can NEVER hold the UI in "stopping" again. In-flight work is handed back safely. - Buttons follow the real state (Start only from stopped; both disabled through the transition), so you can't fight a transition. - Log every Start/Stop button press (routes) and every transition (worker), so the Logs panel shows exactly what each button did. Frontend now trusts s.state (drops the active>0 hack); VERSION → .8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:20:12 -04:00
bvandeusenandClaude Opus 4.8	91ea06be79	feat(agent): Status shows smoothed jobs/min + downloads/min (replace jumpy gauges) ... The buffer / on-GPU / downloader counts flip many times a second, so a 3s status poll only ever samples noise — the tiles looked frozen (same value twice) or random (wildly different), reading as "the Status section doesn't update" when the backend was in fact live (operator-flagged 2026-07-01). Replace the three instantaneous gauge tiles with two derived RATE tiles: - jobs / min — GPU throughput, from the monotonic `processed` counter - downloads / min — fetch throughput, from a new monotonic `downloaded` counter (bumped when a job is decoded into the buffer) Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped at 0 (agent restart resets the counters), and skip a backgrounded-tab gap. The still-useful instantaneous state is demoted, not lost: buffer stays as the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited out` (transient) gets promoted to a tile. backend: worker.status() gains `downloaded`; `_bump(downloaded=)`. frontend: retiled Status + rate math in applyStatus; VERSION → .7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:01:29 -04:00
bvandeusenandClaude Opus 4.8	98b2ac90dd	refactor(agent): DRY pass on the GPU agent worker package ... Consolidate genuine duplication in agent/fc_agent into single-source helpers (behavior-preserving; DRY Pass process #594): worker.py - _fail(jid, image_id, exc, verb) — 4 terminal "fail this job" blocks (downloader HTTP-fault + decode, consumer non-transient + generic). - _release(job_ids) (was _release_owned) — the one lease hand-back path; 6 inline release([jid])+unhold sites now route through it. - _stopped(stop_evt) + _abort_if_stopped(jid, stop_evt) — 4 stop-check -and-release blocks and every bare stop-check. - _timed(stage) contextmanager — ~8 monotonic()/_record() timing pairs; records only on clean exit, matching the old skip-on-raise behavior. - _ewma(prev, x, alpha) module fn — 3 EWMA updates in the autoscaler. client.py - _submit(path, payload) — submit / submit_embedding (retrying session). - _post_quiet(path, payload) — heartbeat / fail / release fire-and-forget. detectors.py - Proposers._top(detector, image, cap) — merges components() and panels(). config.py - _bool_env(name, default) — auto_start / auto_scale env parsing. Left alone (recorded): the xyxy→norm-xywh conversion duplicated across models.py/detectors.py (2 copies, independent wrapper modules — sharing would couple them), and the _ensure_embedder/_ensure_proposers pair (same lock shape, different concepts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 14:53:58 -04:00
bvandeusenandClaude Opus 4.8	8a0237eeea	fix(agent): stream videos via ffmpeg-from-URL instead of downloading the whole file ... The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE file into memory (r.content) just to sample a few frames, which buffered ~1GB in RAM and — on any slow/contended media store — got cut off mid-download (ChunkedEncodingError), failed, and re-leased forever. Measured the media read at ~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak helps; the file simply shouldn't be fully downloaded. Environment-agnostic fix (works for any deployment, completes even when slow): - media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL. It Range-reads only the video index + up to max_frames of content — never the whole file — and reconnect flags resume a dropped transfer instead of failing. Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over speed. Removes the bytes-based sample_frames (dead once videos stream). - worker._download_decode: videos now stream (no fetch_image, no RAM blowup); stills still download+decode. On an ffmpeg miss, probe curator liveness (client.is_reachable) → fail the job if curator is up (unprocessable file, stops the infinite re-lease) vs release if curator is down (transient, survives a redeploy). Auth header passed so it works whether or not /images is gated. Build marker 2026-07-01.6. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 14:15:22 -04:00
bvandeusenandClaude Opus 4.8	aa0605585b	fix(agent): log the REAL fetch/submit failure reason, not "curator unreachable" ... "curator unreachable" was printed for every transient error, hiding whether a single file's transfer stalled (ReadTimeout — curator is up, that stream is slow) or curator itself is down (ConnectTimeout/ConnectionError) or errored (HTTP 5xx). Those need completely different fixes, and we've been diagnosing the download slowness blind. Add _transient_reason(exc) → a specific label (HTTP <code>, else the exception class: ReadTimeout / ConnectTimeout / ConnectionError / …) and use it in both transient paths: - downloader: "fetch failed job <id> (image <id>, ReadTimeout) — released, backing off" - consumer: "submit failed job <id> (<reason>) — released, re-lease later" Now the logs say which failure it actually is (and which image), so we can tell a slow/stalled transfer apart from an unreachable curator. Build marker 2026-07-01.5. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 12:33:51 -04:00
bvandeusenandClaude Opus 4.8	ccbb5cbc9e	fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse) ... Operator hit an outage after the machine slept overnight: the agent showed "curator unreachable" in a loop while curator's API (lease) was actually fine and the browser could still load images — just slowly. Root cause is a feedback loop in the new pipeline: every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for that reason — so the agent piled on more concurrent large-file GETs, saturating curator's web workers + NFS, which slowed curator (and its browser) further and produced more failures → more downloaders. Classic congestion collapse. - Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing — the empty buffer is caused by failures, not the GPU starving. It ramps back up only once downloads succeed again. - DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving path is too many; 8 keeps a fast GPU fed without stampeding curator. - fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a large-but-flowing download still completes, but a stuck/dead connection fails in 60s instead of hanging a downloader for 3 min and piling up stuck requests. Build marker 2026-07-01.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 10:33:18 -04:00
bvandeusenandClaude Opus 4.8	7cdce0c474	feat(agent): temporal video dedup — drop near-duplicate frames before the GPU ... Near-static videos are the dominant GPU load: sampled into up to 64 frames, each re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU perceptual-hash frame dedup upstream of the GPU so the redundant frames are never processed at all (not just their embeds). - media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy keep — a frame survives only if its hash differs from every kept frame by >= min_distance bits (Hamming). A static run collapses to one frame; genuinely distinct scenes all survive. Order + frame_time preserved. - Called in worker._download_decode right after sample_frames, so it runs in the decode stage on the downloader thread (CPU) — the GPU consumers only ever see deduped frames, and buffered video items shrink (less RAM too). - Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M` when it drops any, so video load reduction is visible. Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal axis. Build marker 2026-07-01.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 00:35:03 -04:00
bvandeusenandClaude Opus 4.8	eaae896858	feat(agent): dedupe near-duplicate crops before the SigLIP embed ... Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes, but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept ∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping booru head classes on one head), embedding the same region twice and burning a slot against max_regions. Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass over the pending (crop, template) list right before embed_batch — drop boxes that overlap ≥ iou within the same kind, keep the highest score. The high threshold is deliberate: it collapses only true near-identical boxes while preserving intentional nested crops across scopes (a whole figure vs a small head component sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU (≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 00:20:40 -04:00
bvandeusenandClaude Opus 4.8	afef95a87d	feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash ... The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms), so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit) left the fast GPU idle during every download. Rearchitect worker.py into a producer/consumer pipeline: downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU consumers (detect+embed→submit) - Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a 2nd consumer if it has util/VRAM headroom and lifts throughput, else trim). - Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping RAM + lease look-ahead. VRAM pressure sheds a consumer immediately. - Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU; curator's 180s TTL would otherwise reclaim them mid-buffer). - Preserves all resilience: lease exp-backoff, submit-path retry (#169), release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools and releases every held lease at once (single held-set as source of truth). - Consumers SHARE one embedder + proposers instance (a 2nd consumer adds concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots). - UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU · processed · errors, a buffer-occupancy meter, and a consumers/waited-out line; the dial now tunes downloaders. Build marker 2026-07-01.1. Also fix the operator-flagged detector warning: yolo11n + the comic-panel model threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load- time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3 crop proposers and spamming the log per image. Disable that fusion (unfused inference is correct, marginally slower) and permanently self-disable a proposer on the first inference failure instead of re-throwing forever. Refs milestone 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 23:34:12 -04:00
bvandeusenandClaude Opus 4.8	83f1070a11	fix(agent): bound video GPU work — early-exit the frame loop at max_regions ... Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the SUBMIT — the GPU work was already spent. Break out of the frame loop once accumulated regions reach max_regions, so a long video costs ~a few frames of GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it mean-pools all frames and returns before this loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:52:32 -04:00
bvandeusenandClaude Opus 4.8	c587ac667c	fix(agent): cap figures + global region cap + reset active on stop ... Three safety/robustness fixes from the operator's run logs: - Cap figures per frame (MAX_FIGURES, default 8) like components/panels already are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept (image 81602 looped on 413). This is the acute fix. - Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still exceed the cap (long video), keep the highest-scoring and log the drop, so a submit body can never blow past curator's limit. - Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the reset can't drive it negative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:42:50 -04:00
bvandeusenandClaude Opus 4.8	7e74fa767c	fix(agent): load huge images — disable PIL decompression-bomb guard ... Trusted local library, not an upload surface, so a legitimately large image (90–95M px, operator-flagged) must load. PIL only WARNS at the 89M-px default but RAISES DecompressionBombError at ~179M px, which would fail those jobs. Set Image.MAX_IMAGE_PIXELS = None. (The agent works off individual extracted files — curator's archive_extractor unpacks zip/cbz/rar/7z at import — so this is about big single images, not archives.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:32:51 -04:00
bvandeusenandClaude Opus 4.8	9f1148b110	chore(agent): modernise base → Ubuntu 24.04 / Python 3.12 / CUDA 12.9 ... Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10, CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04: - Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10. - CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13: ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open "Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call, not a hardware limit. - PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed (PEP 668); a single-purpose container owns its environment, so global installs are fine and simplest. - agent/ruff.toml pinned to py312 (was py310) so CI lints against the real runtime; from __future__ import annotations stays (PEP 649 lazy annotations are 3.14, so self-refs still evaluate on 3.12). CI builds the image but has no GPU — validate on the desktop after pull that it starts and loads CUDAExecutionProvider (not CPU fallback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:25:46 -04:00
bvandeusenandClaude Opus 4.8	f01b59f390	fix(agent): py3.10 startup crash + submit-path retry; pin agent ruff to py310 ... The agent container (CUDA base, Python 3.10) crashed on startup with `NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the `from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14 (PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10. CI lint/compile run on 3.14, so it slipped through. - config.py: `from __future__ import annotations` so the self-referential annotation is a string, never evaluated — works on 3.10 and every version. - agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime) and inherit the root rules. Ruff now flags exactly this class as F821, so CI's lint lane catches it instead of shipping a broken image. (CI otherwise lints on 3.14, masking 3.10 issues.) - client.py: submit path now retries in-place. A dedicated session with a urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a momentary blip after the GPU work is done doesn't discard it and force a full re-download + recompute elsewhere. A duplicate submit after a lost response is a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:00:10 -04:00
bvandeusenandClaude Opus 4.8	79269da802	fix(agent): prompt stop + lazy curator polling + build marker; add agent to CI ... Addresses operator reports: Stop never finishes, the agent polls curator constantly, and stale-cached pages get mistaken for a failed deploy. - Stop is prompt: flip _running BEFORE any lock so /status + worker loops see "stopped" immediately, and add a stop/shrink checkpoint in _process (after decode, before the expensive detect+embed) that releases the job and bails — so a Stop doesn't wait out heavy GPU work. - Lazy curator polling: the queue snapshot is fetched only while a browser is actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence, not a constant background loop. The work loop's own lease/submit is curator's only visitor otherwise — nothing polls just to poll. - Build marker: VERSION is embedded in the page and reported on /status; the UI shows a "reload" banner when they differ, so a browser-cached page can't be mistaken for "the new image didn't deploy" (complements the no-store header). CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the GPU agent is linted + syntax-checked before its image builds (build.yml only `docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 21:39:00 -04:00
bvandeusenandClaude Opus 4.8	e6a7fe7d03	feat(agent): per-stage timing breakdown (lease/download/decode/gpu/submit) ... Instrument the job pipeline so we can see where wall-clock actually goes and decide — on data, not theory — whether a download/compute split is worth building. Each stage is timed per job and a rolling breakdown is logged every 30s to the agent console, e.g.: timing/30s — lease 8ms · download 310ms · decode 40ms · gpu 165ms · submit 70ms | wall/job 585ms (214 jobs) - lease timed around client.lease() in the slot loop (per batch). - download = fetch_image; decode = image/frame decode; gpu = detect + CCIP + batched embed; submit = the results POST. One-time model load is excluded from the gpu figure. - Thread-safe accumulator (stage -> [sum, count]) summarised + reset by a small daemon reporter thread; logs only when there was work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 21:28:46 -04:00
bvandeusenandClaude Opus 4.8	f0f031782d	fix(agent): unfreeze status view + smoothed throughput-aware autoscaler + log pane ... Operator: the status tiles (state/active/processed) and the Start/Stop buttons freeze while the GPU meters stay live. Root cause: /status made an INLINE blocking curator call (queue_status) on every poll, and with curator buried under a 112k-job backlog that call stalled — freezing the whole status refresh (the GPU bars survived because /gpu is a lock-free local read). Made worse by the old util-band autoscaler, which grew workers toward the 32 cap forever because util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold — piling load onto curator and the agent process. - /status is now a pure in-memory read: worker.status() is lock-free, and the curator queue snapshot is refreshed by a background poller (never inline). - Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES: samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every ~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a grow stops helping it backs off one and holds, re-probing occasionally. No runaway, no flopping. - GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth, exposed on /gpu) when running, else smoothed client-side — so it glides instead of bouncing 0↔99. - act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the now-always-fast /status refresh recovers state regardless. - Log pane: bound the page to the viewport (height:100vh) so the Logs card scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at 400 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 20:20:14 -04:00
bvandeusenandClaude Opus 4.8	82e1a4e127	fix(agent): send Cache-Control: no-store so a new image isn't masked by cache ... The control page is a static string served with no cache headers, so after pulling a fresh agent image the browser kept showing the OLD UI until a hard refresh (operator-flagged). Add a no-store middleware covering the page and the status/gpu/logs polls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 20:00:26 -04:00
bvandeusenandClaude Opus 4.8	c2e9157822	feat(agent): graceful Start/Stop with starting/stopping states + instant status ... Operator: the buttons fire but the status view doesn't reflect the change. Cause: act() ignored the POST's own status response and waited on the separate /status poll (which lags behind the curator queue call). Now: - act() applies the POST's returned status immediately for instant feedback, and shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled) the moment it's clicked. - A stop that still has in-flight jobs draining shows "stopping" until active hits 0, then resolves to "stopped" on its own. - applyStatus() guards the /status-only fields (connection pill + queue) so the lean action response can't blank them — the Start/Stop path deliberately skips the slow curator call to stay snappy. Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll + autoscaler + /status share a single subprocess instead of piling up in the server thread pool (which was what made clicks feel dead under load). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 19:38:37 -04:00
bvandeusenandClaude Opus 4.8	3b34230fbd	fix(agent): stable util-band autoscaler + live GPU meters ... Two operator-reported issues with the GPU agent: 1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb probed +1, judged it over a too-short noisy throughput window, saw no clear gain and reverted -1 — every tick. Replace it with a GPU-utilization-band controller: HOLD while smoothed util sits in a healthy band, grow only on clear spare capacity (util below the low mark + VRAM headroom), shrink under saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool. Load stays consistent instead of probe/reverting. 2. GPU util/VRAM bars only updated on manual refresh. They rode the /status poll, which blocks on the curator queue call (slow when curator is busy), so the meters froze between refreshes. Give them a dedicated /gpu endpoint (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop the curator queue-status timeout 15s -> 5s so /status itself stays snappy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 19:16:17 -04:00
bvandeusenandClaude Opus 4.8	c259d03618	fix(agent): revert full-width page, grow the Logs section to the bottom ... Operator meant the LOG section should fill down the viewport (vertical), not the whole page going full-width horizontally. Restore the centered column (820px), make .wrap a full-height flex column, and let the Logs card flex to fill the remaining height to the bottom (drop the fixed 230px log-pane cap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 19:09:11 -04:00
bvandeusenandClaude Opus 4.8	2713c3f773	perf(agent): batch SigLIP crop embeds per image + load truncated images ... Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s, truncated-image failures): - BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo components + panels) and embed them in ONE forward pass instead of one forward+lock per crop. The per-crop path serialised every crop through the inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating); batching gives a real GPU-bound workload + far higher throughput. CCIP still runs per figure inline. - LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly- truncated scraped images now load instead of failing the job 3× then erroring ("image file is truncated (N bytes not processed)"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 18:47:33 -04:00
bvandeusenandClaude Opus 4.8	9eaefac385	feat(agent): full-width control page, Copy-logs button, quiet HTTP log noise ... - Page fills the viewport horizontally (drop the 780px cap). - Copy button on the Logs card → copies the console (clipboard API on localhost, textarea-execCommand fallback), with a brief "Copied" confirmation. - Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/ ultralytics to WARNING so the console shows agent activity (detector loads, job errors, autoscale moves) instead of per-request HF-download spam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 18:41:49 -04:00
bvandeusenandClaude Opus 4.8	c1b099e5a3	feat(agent): in-UI log console + a real styling pass on the control page ... - logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root logger; GET /logs serves it; the control page polls it into a console pane — so runs are monitorable without `docker logs`. worker now logs autoscale moves (one line per change, with jobs/s + util + VRAM) and job failures (job + image + reason); detectors already log load/disable. - Restyled the whole control page: a proper dark layout with a header + live connection pill, cards (Control / Status / Logs), a styled Auto switch + worker stepper, status tiles, separate GPU-util and VRAM meters, and the log console. No longer feels like an afterthought; all the existing control hooks are preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 18:34:22 -04:00
bvandeusenandClaude Opus 4.8	6d7b17b0b5	feat(agent): autoscale the worker count (throughput hill-climb), Auto default-on ... The new per-job workload (3 detectors + several SigLIP embeds) is far more GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and is hard to guess. Add an Auto mode (default ON) that finds it: - _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the target: grow while throughput keeps improving and VRAM stays under budget, revert a step that doesn't help, back off under memory pressure (VRAM >= 90%), then settle and periodically re-probe (the GPU/IO balance shifts over a run). - A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the control UI re-enables it. status() reports `auto`; the dial reflects the auto-chosen count (read-only) while Auto is on. - AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 18:19:15 -04:00
bvandeusenandClaude Opus 4.8	ce5db5caaf	chore(agent): default the anatomy + panel proposer weights to working values ... booru_yolo yolov11m_aa22.pt (40MB) + mosesb/best-comic-panel-detection::best.pt. Each self-disables if its download fails, so defaulting them on is safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 15:28:50 -04:00
bvandeusenandClaude Opus 4.8	d5f29f7056	feat(agent): crop proposers — booru_yolo anatomy + COCO person + comic panels (#1202) ... Better region PROPOSERS feeding the existing crop→SigLIP→max-over-bag heads (no change to the learned-tagging approach; no per-tag cost — propose once, embed each region, all heads in one matmul). - detectors.py: lazy ultralytics YOLO wrapper, each proposer independently optional + guarded (a bad weight spec / inference error self-disables that one, logged, never breaks the worker). Weights resolve from an ultralytics name | http(s) URL | "hf_repo::file", cached under HF_HOME. NMS merge so a figure two detectors both find collapses to one crop. - worker: figure boxes = imgutils detect_person ∪ general COCO person (merged) → CCIP + concept (anime + Western/realistic coverage); booru_yolo anatomy components (head/cat-head/anatomy/…) → concept crops; comic panels → kind= 'panel' concept crops. Capped per frame (MAX_COMPONENTS/MAX_PANELS). - config + compose: PERSON_WEIGHTS (default yolo11n.pt, works OOB), ANATOMY_WEIGHTS + PANEL_WEIGHTS (operator sets booru_yolo URL + mosesb panel hf::file; empty = off). ultralytics added to requirements. - backend: image_region 'kind' doc notes 'panel'; no migration (free String, and the bag scorer keys on a non-null siglip_embedding, not the kind, so any SigLIP region joins the bag automatically). Agent is outside CI — py-compiled here; operator tests on the GPU and checks Western-vs-anime crop quality via /api/ccip observability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 15:27:26 -04:00
bvandeusenandClaude Opus 4.8	4daa3f2790	feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190) ... Make the SigLIP embedder an operator choice (drop-in to SigLIP 2: google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no schema change, better small-cue fidelity). A swap = set model + re-embed + retrain, all operator-driven; the GPU agent does the re-embed so it's fast. - settings: embedder_model_name is now a setting (migration 0065) alongside the existing embedder_model_version; both editable + validated (non-empty) in the ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model, model-agnostic), preferring the pre-downloaded local dir for the default so existing deploys don't re-download; rebuilds on a name change. - agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames) under the lease-announced model → POST /jobs/submit_embedding writes image_record.siglip_embedding + siglip_model_version. The lease now announces the model FROM THE SETTING (not a constant). - re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale- version images; 'siglip' now re-embeds concept crops whose version != current (so a swap re-triggers crops, not just the never-embedded back-catalogue). The CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't churn the library at 512px) — the GPU agent owns version re-embeds. Daily 'embed' + 'siglip' beats self-heal. - scoring: score_image only bags embeddings in the CURRENT model's space (whole- image gated by siglip_model_version, concept regions by embedding_version) so a mid-swap stale vector isn't scored by new-space heads; legacy NULL = current. - UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and "Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2. Tests: lease announces model + submit_embedding round-trip; enqueue 'embed' selects stale/unembedded; stale-version excluded from scoring; embedder model settable + empty rejected; siglip gate updated to current-version concept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 10:24:30 -04:00
bvandeusenandClaude Opus 4.8	55fa4656ff	feat(agent): survive + auto-recover when curator is unreachable ... For redeploying curator while away with nobody to restart the agent: - _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx, 401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job fault. On a transport error it hands the job back (best effort) and signals the loop to back off — instead of calling fail(), which would burn the job's server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't re-lease forever. - lease loop retries with capped exponential backoff (poll_idle → 60s) and resets on the first successful lease, so a long outage is gentle and recovery is automatic within ≤60s of curator returning. Sleeps are interruptible so Stop / pool-shrink stays responsive. - AUTO_START env (default on in compose) resumes the worker on container start, so a host reboot / crash-restart (restart: unless-stopped) self-heals with nobody at the desktop. - control UI shows a "waited out" counter + an "curator unreachable, holding work" banner so the recovering state reads as recovery, not failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 08:33:33 -04:00
bvandeusenandClaude Opus 4.8	c6f38b0dac	feat(tagging): SigLIP concept crops + max-over-bag scoring (#114) ... Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray, lactation) that the whole-image SigLIP vector washes out: the GPU agent now embeds figure crops with SigLIP too, stored as kind='concept' regions, and the suggestion rail scores each image as a BAG (whole-image + every concept crop), taking each head's MAX over the bag. The whole-image vector is always in the bag, so this can never score lower than before. Model-agnostic by construction: the server ANNOUNCES the embedding model (HF name + version) in the lease, so the agent loads whatever the heads were trained in and stays in lock-step — a model swap is a server setting + a re-embed migration, never an agent change. - agent: model-agnostic CropEmbedder (torch/transformers get_image_features, fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the back-catalogue backfill never churns figure/CCIP regions; torch cu124 + transformers in the image. - server: lease announces embed_model_name/embed_version; score_image is max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill 'siglip' gates on a missing concept region (drains the back-catalogue, retries failures, no double-enqueue); daily siglip-backfill beat; UI button; /api/ccip/overview reports images_with_concept_siglip. - v1 scope: suggestion rail only — auto-apply stays whole-image (conservative; heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 08:17:47 -04:00
bvandeusenandClaude Opus 4.8	b7fd69815e	feat(agent): raise worker cap to 32 + size the HTTP pool for it (#114) ... At 8 workers the GPU sat at ~5% util / <5GB VRAM — the pipeline is I/O-bound (downloading + decoding images over HTTP), so the GPU starves until many workers overlap that I/O. Raise MAX_CONCURRENCY 8→32 and make the UI worker control a number input (reaching 32 by ±1 was tedious); the cap is reported via /status so the UI clamps to it. Also size the shared requests pool (pool_maxsize=64) — the default 10 would have throttled 32 workers + spammed "connection pool is full". Verified by running; watch GPU util/VRAM climb as you dial up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 19:41:52 -04:00
bvandeusenandClaude Opus 4.8	3abbe58450	fix(agent): flatten transparency onto white before RGB (#114) ... A naive convert('RGB') on a palette-with-transparency image (common: character PNGs on a clear background) lets PIL guess the transparent pixels — black-ish artifacts that bleed into the crop + the CCIP embedding (and the "should be converted to RGBA" warning). to_rgb() composites over white first for a clean, consistent background; used by both stills and video frames. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 19:18:24 -04:00
bvandeusenandClaude Opus 4.8	4a1a9ec5a7	feat(agent): GPU load readout + live worker-count tuning (#114) ... Control UI gains what the operator asked for: - GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how hard the card is working while you're at the desktop. - Worker count is now a live − / + control (POST /concurrency), not just an env: the worker is a pool of independent slots (shared model, so slots add concurrent inference, not N× VRAM). Dial up for speed, down to free the card. Replaces pause/resume with Start/Stop + the worker dial. - Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back via client.release() so they're re-picked immediately (pairs with the server recovery sweep). Not CI-tested (agent/ outside CI) — verified by running. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 19:07:40 -04:00
bvandeusenandClaude Opus 4.8	614b6bc52a	docs(agent): note the NVIDIA Container Toolkit host prereq ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 18:49:41 -04:00
bvandeusenandClaude Opus 4.8	7b10f4caab	fix(agent): cuDNN base image so onnxruntime-gpu loads (#114) ... onnxruntime-gpu needs cuDNN 9; the plain cuda:12.4.1-runtime image lacks it (libcudnn.so.9 missing → CUDAExecutionProvider falls back to CPU). Switch to the -cudnn-runtime variant which bundles cuDNN 9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 18:47:59 -04:00
bvandeusenandClaude Opus 4.8	b6b151a500	docs(agent): docker-compose for the GPU agent ... compose file (pull the published image, GPU reservation, model-cache volume, .env for the token) so the agent runs with `docker compose up -d` instead of a long docker run. A copy + .env template also placed in ~/Documents/fc-gpu-agent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 18:07:33 -04:00
bvandeusenandClaude Opus 4.8	9449241fc2	ci(agent): publish the GPU agent image (build-agent job) ... Build + push fabledcurator-agent alongside web/ml (own CUDA + onnxruntime-gpu image, context=agent/, same tag cadence: main → :main/:latest/:c-<sha>, tag → :<version>). So the operator PULLS + runs it on the GPU machine instead of building locally. README switched to docker pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 14:26:03 -04:00
bvandeusenandClaude Opus 4.8	8419ebd761	feat(agent): desktop GPU agent container — CCIP + figure crops over HTTP (#114) ... The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP (lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres stay private. New top-level agent/ (outside CI scope — verified by running it): - fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/ stop (stop frees the card; unprocessed leases expire + re-queue). - fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed. The two API seams to verify against the installed dghs-imgutils (flagged). - fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence → per-frame instances (the bag). - fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client. - fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress + queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements + README (token → build → run --gpus all → Start; CPU-fallback path). This completes the CCIP pipeline end to end: agent produces region CCIP vectors → RegionService stores → matcher suggests characters → rail. Verified by running on the desktop (not CI). README calls out the imgutils API + model-string checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 14:03:01 -04:00