The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding +
trained head' spine; the operator accepted the tagging system and dropped the
harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint,
tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry,
TagEvalRun model + table (migration 0073), and its tests.
The eval's data loaders + metric helpers were NOT eval-specific — the nightly
heads trainer runs on them — so they moved verbatim to
services/ml/training_data.py (heads.py import updated; behavior unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's
daily embed backfill for the same NULL-embedding images at ~100x the cost
(B1 audit verdict, milestone #124). The backfill TASK stays — the manual
/api/ml/backfill button remains the deliberate CPU fallback pending B3.
- purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on
the live library before removal (A2 audit, milestone #123). Fully retired
per rule 22: tile, store action, route, service fn, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited;
live MB/s dial + net readout in the control UI) is charged by every still
download (streamed chunk reads) and every ffmpeg video stream (metered from
outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget).
Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8
unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's
browser included). Capping the aggregate keeps the desktop usable and still
beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).
One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.
Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
reason instead of another release; strikes never count while stopping, and
clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
GpuJobService so they can't drift) converts expired leases with >=5 lease
grants and pending jobs with >=10 to 'error', preserving the last stored
failure reason. Backstops old agent builds.
Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head
existed, TagEvalCard whenever a persisted run existed — so a fresh Settings
load greeted the operator with several tiles already expanded. All three now
force-open only while their task is actually running (the #877 resurface
behavior on the busy-driven tiles is untouched).
MaintenanceTile additionally persists MANUAL expand/collapse per tile in
localStorage, so the section reloads the way the operator left it; a forced
open while a task runs stays transient and is never saved as a preference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
sample_frames_from_url now returns (frames, reason) — reason carries the
SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found",
or the timeout) instead of only logging it agent-side. The worker folds it
into the failure it reports, so curator's GpuJob.error reads e.g.
no frames sampled from video — ffmpeg exit 183: moov atom not found ...
instead of the bare "(unprocessable)". The errored-jobs list becomes
self-describing: after a retry sweep, surviving errors name their real
defect without needing the agent log. Return-value plumbing (not shared
state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the "no frames sampled from video (unprocessable)" flood
(operator-flagged 2026-07-02, whole 62k-70k image block + others): the
sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4)
frames — which is ZERO for any clip shorter than ~2s. Short animation loops
(0.5s, 1.75s — verified against two originals from different artists) are
complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames,
exited 0, and the agent failed the job as unprocessable. Long videos worked,
so only the short-clip class flooded.
Fix: sample with select ("first frame always, then one per interval of
timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range
yuv420p sources don't trip the mjpeg encoder's full-range strictness
(secondary failure observed on a 4440x2760 clip). Verified locally against
both failing originals (frames extracted, PIL-clean) and a synthetic 15s
video (4 frames at t=0/4/8/12 — long-video behavior unchanged).
Observability (why this hid for weeks): ffmpeg's stderr was discarded, so
every failure logged only "no frames sampled". stderr now goes to a temp
file and its tail is logged on any produced-no-frames/timeout failure — the
log names the actual ffmpeg reason from now on. Also: frames written before
a mid-stream ffmpeg error are now kept (partial > nothing).
VERSION → 2026-07-02.1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
THE root cause of "the Status section doesn't update" (chased across several
rounds; the backend was always healthy). `#capn` (the max-concurrency number)
was nested inside `#conchint`:
<div id=conchint>… · max <b id=capn>8</b></div>
and applyStatus() ran, every call: `capn.textContent=CAP` AND
`conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces
ALL of conchint's children — destroying the <b id=capn> node. So:
call 1: capn exists → tiles update → conchint.textContent DELETES capn
call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) →
applyStatus throws on its FIRST line → aborts before any tile →
frozen.
This is exactly the observed "ticks a couple times then freezes", and why
/gpu + /logs (which never touch capn) kept updating fine.
The capn write was redundant anyway — conchint.textContent already renders
the max. Remove the nested <b id=capn> element and the capn.textContent line;
the hint still shows "· max N". VERSION → .10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two follow-ups from live debugging of "work/min never populates" and
"stopped never reached".
1) jobs/min + downloads/min are now computed in the BACKEND on a fixed
cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were
derived client-side from poll deltas with a dt<30s guard — but a
backgrounded/unfocused browser tab throttles its timers to ~1/min, so
every delta exceeded 30s and the guard blanked the rates forever. A
server-side rate is independent of how often the tab polls. Frontend just
displays s.jobs_per_min / s.downloads_per_min. VERSION → .9.
2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a
slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see
the stop signal until ffmpeg returned, so Stop detached still-running
threads and work kept flowing long after — "stopped" that wasn't really
stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a
`should_stop` callback every 0.5s, terminating (then killing) the process
at once on Stop or the per-video timeout. A stop-killed job is handed back
(transient), not failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Status pill hung on "stopping" forever (operator-flagged 2026-07-01).
Root cause: the backend had no lifecycle state — status() only returned
running/stopped — so the UI FABRICATED "stopping" in JS as `!running &&
active>0`. That pill only cleared when the backend's `active` counter hit 0,
but stop() (a) blocked the HTTP handler on lease-release calls to curator and
(b) left `active>0` whenever a consumer wedged mid-submit/release to an
overloaded curator → "stopping" that never resolved.
Give the backend a real, truthful state it drives itself:
stopped → starting → running → stopping → stopped
- start(): → starting; a downloader flips it to running on its FIRST
successful lease (so "running" means curator is actually answering, not
just "Start was clicked"). If curator's down it honestly stays "starting".
- stop(): → stopping; returns immediately (no handler block). A background
monitor waits for the worker threads to actually exit, releases leases,
then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can
NEVER hold the UI in "stopping" again. In-flight work is handed back safely.
- Buttons follow the real state (Start only from stopped; both disabled
through the transition), so you can't fight a transition.
- Log every Start/Stop button press (routes) and every transition (worker),
so the Logs panel shows exactly what each button did.
Frontend now trusts s.state (drops the active>0 hack); VERSION → .8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The buffer / on-GPU / downloader counts flip many times a second, so a 3s
status poll only ever samples noise — the tiles looked frozen (same value
twice) or random (wildly different), reading as "the Status section doesn't
update" when the backend was in fact live (operator-flagged 2026-07-01).
Replace the three instantaneous gauge tiles with two derived RATE tiles:
- jobs / min — GPU throughput, from the monotonic `processed` counter
- downloads / min — fetch throughput, from a new monotonic `downloaded`
counter (bumped when a job is decoded into the buffer)
Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the
reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped
at 0 (agent restart resets the counters), and skip a backgrounded-tab gap.
The still-useful instantaneous state is demoted, not lost: buffer stays as
the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited
out` (transient) gets promoted to a tile.
backend: worker.status() gains `downloaded`; `_bump(downloaded=)`.
frontend: retiled Status + rate math in applyStatus; VERSION → .7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE
file into memory (r.content) just to sample a few frames, which buffered ~1GB in
RAM and — on any slow/contended media store — got cut off mid-download
(ChunkedEncodingError), failed, and re-leased forever. Measured the media read at
~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak
helps; the file simply shouldn't be fully downloaded.
Environment-agnostic fix (works for any deployment, completes even when slow):
- media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL.
It Range-reads only the video index + up to max_frames of content — never the
whole file — and reconnect flags resume a dropped transfer instead of failing.
Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over
speed. Removes the bytes-based sample_frames (dead once videos stream).
- worker._download_decode: videos now stream (no fetch_image, no RAM blowup);
stills still download+decode. On an ffmpeg miss, probe curator liveness
(client.is_reachable) → fail the job if curator is up (unprocessable file, stops
the infinite re-lease) vs release if curator is down (transient, survives a
redeploy). Auth header passed so it works whether or not /images is gated.
Build marker 2026-07-01.6. Refs issue #1225.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
"curator unreachable" was printed for every transient error, hiding whether a
single file's transfer stalled (ReadTimeout — curator is up, that stream is slow)
or curator itself is down (ConnectTimeout/ConnectionError) or errored (HTTP 5xx).
Those need completely different fixes, and we've been diagnosing the download
slowness blind.
Add _transient_reason(exc) → a specific label (HTTP <code>, else the exception
class: ReadTimeout / ConnectTimeout / ConnectionError / …) and use it in both
transient paths:
- downloader: "fetch failed job <id> (image <id>, ReadTimeout) — released, backing off"
- consumer: "submit failed job <id> (<reason>) — released, re-lease later"
Now the logs say which failure it actually is (and which image), so we can tell a
slow/stalled transfer apart from an unreachable curator. Build marker 2026-07-01.5.
Refs issue #1225.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The image library is on a CIFS/SMB share (mounted rsize=4 MiB, actimeo=1), and
Quart's FileBody streams in 8 KiB chunks — so serving one large original was
~19k network round-trips to the storage server, i.e. 30–58s per download
(operator-flagged). That's what starved the GPU agent (constant "curator
unreachable" backoff) AND slowed the browser: every byte is read off CIFS and
streamed through the Python app (no reverse-proxy sendfile), and only 2 hypercorn
workers meant the agent + the browser's thumbnail grid queued behind each other.
In-container fix, no new service:
- Raise FileBody.buffer_size 8 KiB → 4 MiB in create_app, matching the mount's
read size: one round-trip per read, ~500× fewer. buffer_size is the MAX read so
small thumbnails still read in one gulp, and Range/mime/ETag/conditional
handling lives on Response — all preserved. Guarded so a Quart-internal change
can't break boot.
- HYPERCORN_WORKERS default 2 → 4 so concurrent /images requests stop queuing.
Expected: large-file transfers drop from ~40s toward link speed (a few seconds)
for the agent and the browser. See issue #1223.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The gallery's newest/oldest sort keys off image_record.effective_date =
COALESCE(primary post's post_date, created_at). The primary post is often the
repost/download the file came from, so the grid led with download dates rather
than when content was first posted (operator-flagged).
Add a second materialized sort key, earliest_post_date = MIN(post_date) across
ALL of an image's provenance posts (every post it appears in), else created_at —
the original publish date. Mirrors the effective_date pattern so the sort stays a
forward index scan.
- alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill
created_at baseline then MIN over image_provenance ⋈ post.
- importer: recompute earliest_post_date whenever a dated post is linked (MIN over
the image's provenance, which now includes the just-added row).
- gallery_service: new sorts posted_new / posted_old key off earliest_post_date;
cursor + year/month grouping follow the active column transparently.
- api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads
with original publish date. newest/oldest (effective_date) still available.
- frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post
date); existing effective-date sorts relabelled "Newest/Oldest added".
- tests: service test asserts posted_new/posted_old key off earliest_post_date;
frontend default-sort omission test updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.
- Failure-aware autoscaling: if transient download failures rose since the last
decision, SHRINK the downloader pool toward the floor instead of growing — the
empty buffer is caused by failures, not the GPU starving. It ramps back up only
once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
large-but-flowing download still completes, but a stuck/dead connection fails in
60s instead of hanging a downloader for 3 min and piling up stuck requests.
Build marker 2026-07-01.4.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator wants the Explore "related" rail to span more — the #1188 diversifier
was tuned conservatively. Push all three knobs so it reaches further across
clusters instead of clumping near the anchor:
- MMR lam 0.55 → 0.40 — weight the diversity penalty harder (the main dial).
- candidate pool min(200, max(limit*5, 60)) → min(400, max(limit*8, 100)) — a
wider nearest-cosine pool so MMR has genuinely distinct neighbourhoods to pick
from, not just the near-dupes.
- pHash dup_threshold 6 → 8 — collapse more near-duplicate reposts/clones,
freeing rail slots for distinct picks.
Still deterministic (same set per image, just more spread) and relevance-anchored
via the lam*sim-to-anchor term. Backend-only; no migration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Near-static videos are the dominant GPU load: sampled into up to 64 frames, each
re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU
perceptual-hash frame dedup upstream of the GPU so the redundant frames are never
processed at all (not just their embeds).
- media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy
keep — a frame survives only if its hash differs from every kept frame by
>= min_distance bits (Hamming). A static run collapses to one frame; genuinely
distinct scenes all survive. Order + frame_time preserved.
- Called in worker._download_decode right after sample_frames, so it runs in the
decode stage on the downloader thread (CPU) — the GPU consumers only ever see
deduped frames, and buffered video items shrink (less RAM too).
- Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief
localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M`
when it drops any, so video load reduction is visible.
Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal
axis. Build marker 2026-07-01.3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes,
but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept
∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates
slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping
booru head classes on one head), embedding the same region twice and burning a
slot against max_regions.
Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass
over the pending (crop, template) list right before embed_batch — drop boxes that
overlap ≥ iou within the same kind, keep the highest score. The high threshold is
deliberate: it collapses only true near-identical boxes while preserving
intentional nested crops across scopes (a whole figure vs a small head component
sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU
(≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and
region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms),
so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit)
left the fast GPU idle during every download. Rearchitect worker.py into a
producer/consumer pipeline:
downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU
consumers (detect+embed→submit)
- Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER
count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a
2nd consumer if it has util/VRAM headroom and lifts throughput, else trim).
- Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping
RAM + lease look-ahead. VRAM pressure sheds a consumer immediately.
- Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU;
curator's 180s TTL would otherwise reclaim them mid-buffer).
- Preserves all resilience: lease exp-backoff, submit-path retry (#169),
release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools
and releases every held lease at once (single held-set as source of truth).
- Consumers SHARE one embedder + proposers instance (a 2nd consumer adds
concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots).
- UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU ·
processed · errors, a buffer-occupancy meter, and a consumers/waited-out line;
the dial now tunes downloaders. Build marker 2026-07-01.1.
Also fix the operator-flagged detector warning: yolo11n + the comic-panel model
threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load-
time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3
crop proposers and spamming the log per image. Disable that fusion (unfused
inference is correct, marginally slower) and permanently self-disable a proposer
on the first inference failure instead of re-throwing forever.
Refs milestone 122.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples
up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of
detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the
SUBMIT — the GPU work was already spent. Break out of the frame loop once
accumulated regions reach max_regions, so a long video costs ~a few frames of
GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it
mean-pools all frames and returns before this loop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Three safety/robustness fixes from the operator's run logs:
- Cap figures per frame (MAX_FIGURES, default 8) like components/panels already
are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds
of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept
(image 81602 looped on 413). This is the acute fix.
- Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still
exceed the cap (long video), keep the highest-scoring and log the drop, so a
submit body can never blow past curator's limit.
- Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the
meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the
reset can't drive it negative.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Trusted local library, not an upload surface, so a legitimately large image
(90–95M px, operator-flagged) must load. PIL only WARNS at the 89M-px default but
RAISES DecompressionBombError at ~179M px, which would fail those jobs. Set
Image.MAX_IMAGE_PIXELS = None. (The agent works off individual extracted files —
curator's archive_extractor unpacks zip/cbz/rar/7z at import — so this is about
big single images, not archives.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10,
CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04:
- Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10.
- CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the
default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13:
ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open
"Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The
GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call,
not a hardware limit.
- PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed
(PEP 668); a single-purpose container owns its environment, so global installs
are fine and simplest.
- agent/ruff.toml pinned to py312 (was py310) so CI lints against the real
runtime; from __future__ import annotations stays (PEP 649 lazy annotations
are 3.14, so self-refs still evaluate on 3.12).
CI builds the image but has no GPU — validate on the desktop after pull that it
starts and loads CUDAExecutionProvider (not CPU fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent container (CUDA base, Python 3.10) crashed on startup with
`NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the
`from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14
(PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10.
CI lint/compile run on 3.14, so it slipped through.
- config.py: `from __future__ import annotations` so the self-referential
annotation is a string, never evaluated — works on 3.10 and every version.
- agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime)
and inherit the root rules. Ruff now flags exactly this class as F821, so CI's
lint lane catches it instead of shipping a broken image. (CI otherwise lints on
3.14, masking 3.10 issues.)
- client.py: submit path now retries in-place. A dedicated session with a
urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a
momentary blip after the GPU work is done doesn't discard it and force a full
re-download + recompute elsewhere. A duplicate submit after a lost response is
a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Addresses operator reports: Stop never finishes, the agent polls curator
constantly, and stale-cached pages get mistaken for a failed deploy.
- Stop is prompt: flip _running BEFORE any lock so /status + worker loops see
"stopped" immediately, and add a stop/shrink checkpoint in _process (after
decode, before the expensive detect+embed) that releases the job and bails —
so a Stop doesn't wait out heavy GPU work.
- Lazy curator polling: the queue snapshot is fetched only while a browser is
actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence,
not a constant background loop. The work loop's own lease/submit is curator's
only visitor otherwise — nothing polls just to poll.
- Build marker: VERSION is embedded in the page and reported on /status; the UI
shows a "reload" banner when they differ, so a browser-cached page can't be
mistaken for "the new image didn't deploy" (complements the no-store header).
CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the
GPU agent is linted + syntax-checked before its image builds (build.yml only
`docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
bvandeusen