Commit Graph

1109 Commits

Author SHA1 Message Date
bvandeusen 560a5000a2 Merge pull request 'gallery: sort by earliest post date across all posts (new default)' (#178) from dev into main
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m26s
2026-07-01 10:54:41 -04:00
bvandeusen c22f37d64d feat(gallery): sort by earliest post date across all posts (new default)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m24s
The gallery's newest/oldest sort keys off image_record.effective_date =
COALESCE(primary post's post_date, created_at). The primary post is often the
repost/download the file came from, so the grid led with download dates rather
than when content was first posted (operator-flagged).

Add a second materialized sort key, earliest_post_date = MIN(post_date) across
ALL of an image's provenance posts (every post it appears in), else created_at —
the original publish date. Mirrors the effective_date pattern so the sort stays a
forward index scan.

- alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill
  created_at baseline then MIN over image_provenance ⋈ post.
- importer: recompute earliest_post_date whenever a dated post is linked (MIN over
  the image's provenance, which now includes the just-added row).
- gallery_service: new sorts posted_new / posted_old key off earliest_post_date;
  cursor + year/month grouping follow the active column transparently.
- api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads
  with original publish date. newest/oldest (effective_date) still available.
- frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post
  date); existing effective-date sorts relabelled "Newest/Oldest added".
- tests: service test asserts posted_new/posted_old key off earliest_post_date;
  frontend default-sort omission test updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-07-01 10:46:09 -04:00
bvandeusen 7ddad231f8 Merge pull request 'agent: stop downloader pool stampeding a slow curator (congestion collapse)' (#177) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m22s
2026-07-01 10:45:48 -04:00
bvandeusen ccbb5cbc9e fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s
Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.

- Failure-aware autoscaling: if transient download failures rose since the last
  decision, SHRINK the downloader pool toward the floor instead of growing — the
  empty buffer is caused by failures, not the GPU starving. It ramps back up only
  once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
  path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
  large-but-flowing download still completes, but a stuck/dead connection fails in
  60s instead of hanging a downloader for 3 min and piling up stuck requests.

Build marker 2026-07-01.4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-07-01 10:33:18 -04:00
bvandeusen a78f7eaace Merge pull request 'explore: more variance in the related rail (stronger MMR diversification)' (#176) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
2026-07-01 00:50:59 -04:00
bvandeusen ef3318aac1 feat(explore): more variance in the related rail (stronger MMR diversification)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m24s
Operator wants the Explore "related" rail to span more — the #1188 diversifier
was tuned conservatively. Push all three knobs so it reaches further across
clusters instead of clumping near the anchor:

- MMR lam 0.55 → 0.40 — weight the diversity penalty harder (the main dial).
- candidate pool min(200, max(limit*5, 60)) → min(400, max(limit*8, 100)) — a
  wider nearest-cosine pool so MMR has genuinely distinct neighbourhoods to pick
  from, not just the near-dupes.
- pHash dup_threshold 6 → 8 — collapse more near-duplicate reposts/clones,
  freeing rail slots for distinct picks.

Still deterministic (same set per image, just more spread) and relevance-anchored
via the lam*sim-to-anchor term. Backend-only; no migration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-07-01 00:46:56 -04:00
bvandeusen 71337b0ba4 Merge pull request 'agent: temporal video dedup — drop near-duplicate frames before the GPU' (#175) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
2026-07-01 00:38:53 -04:00
bvandeusen 7cdce0c474 feat(agent): temporal video dedup — drop near-duplicate frames before the GPU
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m24s
Near-static videos are the dominant GPU load: sampled into up to 64 frames, each
re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU
perceptual-hash frame dedup upstream of the GPU so the redundant frames are never
processed at all (not just their embeds).

- media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy
  keep — a frame survives only if its hash differs from every kept frame by
  >= min_distance bits (Hamming). A static run collapses to one frame; genuinely
  distinct scenes all survive. Order + frame_time preserved.
- Called in worker._download_decode right after sample_frames, so it runs in the
  decode stage on the downloader thread (CPU) — the GPU consumers only ever see
  deduped frames, and buffered video items shrink (less RAM too).
- Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief
  localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M`
  when it drops any, so video load reduction is visible.

Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal
axis. Build marker 2026-07-01.3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-07-01 00:35:03 -04:00
bvandeusen 3996205f3b Merge pull request 'agent: dedupe near-duplicate crops before the SigLIP embed' (#174) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m26s
2026-07-01 00:24:39 -04:00
bvandeusen eaae896858 feat(agent): dedupe near-duplicate crops before the SigLIP embed
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes,
but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept
∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates
slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping
booru head classes on one head), embedding the same region twice and burning a
slot against max_regions.

Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass
over the pending (crop, template) list right before embed_batch — drop boxes that
overlap ≥ iou within the same kind, keep the highest score. The high threshold is
deliberate: it collapses only true near-identical boxes while preserving
intentional nested crops across scopes (a whole figure vs a small head component
sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU
(≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and
region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-07-01 00:20:40 -04:00
bvandeusen 9f9db01456 Merge pull request 'agent: download/GPU producer-consumer pipeline + detector fuse fix' (#173) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
2026-06-30 23:39:25 -04:00
bvandeusen afef95a87d feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m25s
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms),
so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit)
left the fast GPU idle during every download. Rearchitect worker.py into a
producer/consumer pipeline:

  downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU
  consumers (detect+embed→submit)

- Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER
  count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a
  2nd consumer if it has util/VRAM headroom and lifts throughput, else trim).
- Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping
  RAM + lease look-ahead. VRAM pressure sheds a consumer immediately.
- Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU;
  curator's 180s TTL would otherwise reclaim them mid-buffer).
- Preserves all resilience: lease exp-backoff, submit-path retry (#169),
  release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools
  and releases every held lease at once (single held-set as source of truth).
- Consumers SHARE one embedder + proposers instance (a 2nd consumer adds
  concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots).
- UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU ·
  processed · errors, a buffer-occupancy meter, and a consumers/waited-out line;
  the dial now tunes downloaders. Build marker 2026-07-01.1.

Also fix the operator-flagged detector warning: yolo11n + the comic-panel model
threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load-
time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3
crop proposers and spamming the log per image. Disable that fusion (unfused
inference is correct, marginally slower) and permanently self-disable a proposer
on the first inference failure instead of re-throwing forever.

Refs milestone 122.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 23:34:12 -04:00
bvandeusen 216b7fc743 Merge pull request 'Agent: bound video GPU work (early-exit frame loop at max_regions)' (#172) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s
2026-06-30 22:57:33 -04:00
bvandeusen 83f1070a11 fix(agent): bound video GPU work — early-exit the frame loop at max_regions
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m23s
Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples
up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of
detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the
SUBMIT — the GPU work was already spent. Break out of the frame loop once
accumulated regions reach max_regions, so a long video costs ~a few frames of
GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it
mean-pools all frames and returns before this loop).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 22:52:32 -04:00
bvandeusen fbb76e6f36 Merge pull request 'Agent: huge-image load + figure/region caps + stale-active fix' (#171) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
2026-06-30 22:47:48 -04:00
bvandeusen c587ac667c fix(agent): cap figures + global region cap + reset active on stop
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m25s
Three safety/robustness fixes from the operator's run logs:

- Cap figures per frame (MAX_FIGURES, default 8) like components/panels already
  are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds
  of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept
  (image 81602 looped on 413). This is the acute fix.
- Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still
  exceed the cap (long video), keep the highest-scoring and log the drop, so a
  submit body can never blow past curator's limit.
- Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the
  meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the
  reset can't drive it negative.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 22:42:50 -04:00
bvandeusen 7e74fa767c fix(agent): load huge images — disable PIL decompression-bomb guard
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m22s
Trusted local library, not an upload surface, so a legitimately large image
(90–95M px, operator-flagged) must load. PIL only WARNS at the 89M-px default but
RAISES DecompressionBombError at ~179M px, which would fail those jobs. Set
Image.MAX_IMAGE_PIXELS = None. (The agent works off individual extracted files —
curator's archive_extractor unpacks zip/cbz/rar/7z at import — so this is about
big single images, not archives.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 22:32:51 -04:00
bvandeusen 5ef1478ade Merge pull request 'Modernise agent base → Ubuntu 24.04 / Python 3.12 / CUDA 12.9' (#170) from dev into main
Build images / build-ml (push) Failing after 3s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 41s
CI / integration (push) Successful in 3m22s
Build images / build-agent (push) Successful in 8m44s
2026-06-30 22:30:18 -04:00
bvandeusen 9f1148b110 chore(agent): modernise base → Ubuntu 24.04 / Python 3.12 / CUDA 12.9
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10,
CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04:

- Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10.
- CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the
  default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13:
  ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open
  "Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The
  GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call,
  not a hardware limit.
- PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed
  (PEP 668); a single-purpose container owns its environment, so global installs
  are fine and simplest.
- agent/ruff.toml pinned to py312 (was py310) so CI lints against the real
  runtime; from __future__ import annotations stays (PEP 649 lazy annotations
  are 3.14, so self-refs still evaluate on 3.12).

CI builds the image but has no GPU — validate on the desktop after pull that it
starts and loads CUDAExecutionProvider (not CPU fallback).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 22:25:46 -04:00
bvandeusen 88ff4147e1 Merge pull request 'Fix: agent py3.10 startup crash + submit-path retry + pin agent ruff to py310' (#169) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
2026-06-30 22:03:57 -04:00
bvandeusen f01b59f390 fix(agent): py3.10 startup crash + submit-path retry; pin agent ruff to py310
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m24s
The agent container (CUDA base, Python 3.10) crashed on startup with
`NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the
`from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14
(PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10.
CI lint/compile run on 3.14, so it slipped through.

- config.py: `from __future__ import annotations` so the self-referential
  annotation is a string, never evaluated — works on 3.10 and every version.
- agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime)
  and inherit the root rules. Ruff now flags exactly this class as F821, so CI's
  lint lane catches it instead of shipping a broken image. (CI otherwise lints on
  3.14, masking 3.10 issues.)
- client.py: submit path now retries in-place. A dedicated session with a
  urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a
  momentary blip after the GPU work is done doesn't discard it and force a full
  re-download + recompute elsewhere. A duplicate submit after a lost response is
  a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 22:00:10 -04:00
bvandeusen 1ea02ad44c Merge pull request 'Release: prompt agent stop + lazy curator polling + build marker + agent in CI' (#168) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m23s
2026-06-30 21:42:55 -04:00
bvandeusen 79269da802 fix(agent): prompt stop + lazy curator polling + build marker; add agent to CI
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
Addresses operator reports: Stop never finishes, the agent polls curator
constantly, and stale-cached pages get mistaken for a failed deploy.

- Stop is prompt: flip _running BEFORE any lock so /status + worker loops see
  "stopped" immediately, and add a stop/shrink checkpoint in _process (after
  decode, before the expensive detect+embed) that releases the job and bails —
  so a Stop doesn't wait out heavy GPU work.
- Lazy curator polling: the queue snapshot is fetched only while a browser is
  actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence,
  not a constant background loop. The work loop's own lease/submit is curator's
  only visitor otherwise — nothing polls just to poll.
- Build marker: VERSION is embedded in the page and reported on /status; the UI
  shows a "reload" banner when they differ, so a browser-cached page can't be
  mistaken for "the new image didn't deploy" (complements the no-store header).

CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the
GPU agent is linted + syntax-checked before its image builds (build.yml only
`docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 21:39:00 -04:00
bvandeusen 937cfb65b4 Merge pull request 'Release: agent per-stage timing breakdown' (#167) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 5s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m23s
2026-06-30 21:28:57 -04:00
bvandeusen e6a7fe7d03 feat(agent): per-stage timing breakdown (lease/download/decode/gpu/submit)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m24s
Instrument the job pipeline so we can see where wall-clock actually goes and
decide — on data, not theory — whether a download/compute split is worth
building. Each stage is timed per job and a rolling breakdown is logged every
30s to the agent console, e.g.:

  timing/30s — lease 8ms · download 310ms · decode 40ms · gpu 165ms · submit 70ms | wall/job 585ms (214 jobs)

- lease timed around client.lease() in the slot loop (per batch).
- download = fetch_image; decode = image/frame decode; gpu = detect + CCIP +
  batched embed; submit = the results POST. One-time model load is excluded
  from the gpu figure.
- Thread-safe accumulator (stage -> [sum, count]) summarised + reset by a small
  daemon reporter thread; logs only when there was work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 21:28:46 -04:00
bvandeusen baac851220 Merge pull request 'perf: GPU-job leasing stays O(batch) — partial indexes + two-phase lease' (#166) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-agent (push) Successful in 9s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m27s
2026-06-30 21:18:23 -04:00
bvandeusen 181f1c6a27 perf(gpu-queue): partial indexes + two-phase lease so leasing stays O(batch)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
The throughput bottleneck was curator-side, not the network. lease() claimed the
lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a
plain `status` index Postgres walked the primary key from id=1, skipping the
entire prefix of already done/error rows before reaching pending ones. As `done`
grew (69k+), every lease became an O(done) scan — leasing crawled, the DB
saturated, and even /status (the queue GROUP BY count) stalled the agent.

- Migration 0070 adds two partial indexes over just the live slice: pending rows
  indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery
  + orphan sweep). They stay tiny no matter how large the done/error history.
- lease() split into two phases so each uses a partial index: claim pending
  first (id-ordered, O(batch)); reclaim expired leases only when pending can't
  fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim).
- Model __table_args__ declares the indexes so ORM and schema agree.
- Test: a done-prefix at low ids must not stop the lease reaching pending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 21:12:12 -04:00
bvandeusen a28c33281a Merge pull request 'Release: unfreeze agent status view + smoothed throughput autoscaler + log pane' (#165) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 9s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m23s
2026-06-30 20:20:34 -04:00
bvandeusen f0f031782d fix(agent): unfreeze status view + smoothed throughput-aware autoscaler + log pane
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m23s
Operator: the status tiles (state/active/processed) and the Start/Stop buttons
freeze while the GPU meters stay live. Root cause: /status made an INLINE
blocking curator call (queue_status) on every poll, and with curator buried
under a 112k-job backlog that call stalled — freezing the whole status refresh
(the GPU bars survived because /gpu is a lock-free local read). Made worse by the
old util-band autoscaler, which grew workers toward the 32 cap forever because
util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold —
piling load onto curator and the agent process.

- /status is now a pure in-memory read: worker.status() is lock-free, and the
  curator queue snapshot is refreshed by a background poller (never inline).
- Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES:
  samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every
  ~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a
  grow stops helping it backs off one and holds, re-probing occasionally. No
  runaway, no flopping.
- GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth,
  exposed on /gpu) when running, else smoothed client-side — so it glides
  instead of bouncing 0↔99.
- act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the
  now-always-fast /status refresh recovers state regardless.
- Log pane: bound the page to the viewport (height:100vh) so the Logs card
  scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at
  400 lines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 20:20:14 -04:00
bvandeusen 40be0a9323 Merge pull request 'Release: agent page no-store cache header' (#164) from dev into main
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Failing after 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m23s
2026-06-30 20:00:36 -04:00
bvandeusen 82e1a4e127 fix(agent): send Cache-Control: no-store so a new image isn't masked by cache
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m21s
The control page is a static string served with no cache headers, so after
pulling a fresh agent image the browser kept showing the OLD UI until a hard
refresh (operator-flagged). Add a no-store middleware covering the page and the
status/gpu/logs polls.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 20:00:26 -04:00
bvandeusen 2c6bf26bfc Merge pull request 'Release: graceful Start/Stop (starting/stopping states) + cached GPU reads' (#163) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m24s
2026-06-30 19:38:50 -04:00
bvandeusen c2e9157822 feat(agent): graceful Start/Stop with starting/stopping states + instant status
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m22s
Operator: the buttons fire but the status view doesn't reflect the change. Cause:
act() ignored the POST's own status response and waited on the separate /status
poll (which lags behind the curator queue call). Now:

- act() applies the POST's returned status immediately for instant feedback, and
  shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled)
  the moment it's clicked.
- A stop that still has in-flight jobs draining shows "stopping" until active
  hits 0, then resolves to "stopped" on its own.
- applyStatus() guards the /status-only fields (connection pill + queue) so the
  lean action response can't blank them — the Start/Stop path deliberately skips
  the slow curator call to stay snappy.

Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a
time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll +
autoscaler + /status share a single subprocess instead of piling up in the
server thread pool (which was what made clicks feel dead under load).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 19:38:37 -04:00
bvandeusen 3c1e76bd44 Merge pull request 'Release: stable util-band autoscaler + live GPU meters' (#162) from dev into main
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 5s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
Build images / build-web (push) Successful in 6s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m44s
2026-06-30 19:16:29 -04:00
bvandeusen 3b34230fbd fix(agent): stable util-band autoscaler + live GPU meters
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m40s
Two operator-reported issues with the GPU agent:

1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb
   probed +1, judged it over a too-short noisy throughput window, saw no clear
   gain and reverted -1 — every tick. Replace it with a GPU-utilization-band
   controller: HOLD while smoothed util sits in a healthy band, grow only on
   clear spare capacity (util below the low mark + VRAM headroom), shrink under
   saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced
   (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool.
   Load stays consistent instead of probe/reverting.

2. GPU util/VRAM bars only updated on manual refresh. They rode the /status
   poll, which blocks on the curator queue call (slow when curator is busy), so
   the meters froze between refreshes. Give them a dedicated /gpu endpoint
   (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop
   the curator queue-status timeout 15s -> 5s so /status itself stays snappy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 19:16:17 -04:00
bvandeusen 831c5b1c10 Merge pull request 'Release: agent page — centered column + logs fill to bottom' (#161) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
2026-06-30 19:10:50 -04:00
bvandeusen c259d03618 fix(agent): revert full-width page, grow the Logs section to the bottom
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m24s
Operator meant the LOG section should fill down the viewport (vertical), not the
whole page going full-width horizontally. Restore the centered column (820px),
make .wrap a full-height flex column, and let the Logs card flex to fill the
remaining height to the bottom (drop the fixed 230px log-pane cap).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 19:09:11 -04:00
bvandeusen 76b6af4903 Merge pull request 'Release: agent perf (batched crop embeds) + UI (full-width, copy logs, quiet noise) + truncated-image fix' (#160) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m32s
2026-06-30 18:47:55 -04:00
bvandeusen 2713c3f773 perf(agent): batch SigLIP crop embeds per image + load truncated images
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s
Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s,
truncated-image failures):

- BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo
  components + panels) and embed them in ONE forward pass instead of one
  forward+lock per crop. The per-crop path serialised every crop through the
  inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating);
  batching gives a real GPU-bound workload + far higher throughput. CCIP still
  runs per figure inline.
- LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly-
  truncated scraped images now load instead of failing the job 3× then erroring
  ("image file is truncated (N bytes not processed)").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 18:47:33 -04:00
bvandeusen 9eaefac385 feat(agent): full-width control page, Copy-logs button, quiet HTTP log noise
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
- Page fills the viewport horizontally (drop the 780px cap).
- Copy button on the Logs card → copies the console (clipboard API on localhost,
  textarea-execCommand fallback), with a brief "Copied" confirmation.
- Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/
  ultralytics to WARNING so the console shows agent activity (detector loads,
  job errors, autoscale moves) instead of per-request HF-download spam.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 18:41:49 -04:00
bvandeusen 92494ec4ed Merge pull request 'Release: agent control-page logs console + styling pass' (#159) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s
2026-06-30 18:37:35 -04:00
bvandeusen c1b099e5a3 feat(agent): in-UI log console + a real styling pass on the control page
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m29s
- logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root
  logger; GET /logs serves it; the control page polls it into a console pane —
  so runs are monitorable without `docker logs`. worker now logs autoscale moves
  (one line per change, with jobs/s + util + VRAM) and job failures (job + image
  + reason); detectors already log load/disable.
- Restyled the whole control page: a proper dark layout with a header + live
  connection pill, cards (Control / Status / Logs), a styled Auto switch +
  worker stepper, status tiles, separate GPU-util and VRAM meters, and the log
  console. No longer feels like an afterthought; all the existing control hooks
  are preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 18:34:22 -04:00
bvandeusen bdfc17477c Merge pull request 'Release: GPU agent autoscaler (throughput hill-climb, Auto default-on)' (#158) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m24s
2026-06-30 18:23:15 -04:00
bvandeusen 6d7b17b0b5 feat(agent): autoscale the worker count (throughput hill-climb), Auto default-on
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m23s
The new per-job workload (3 detectors + several SigLIP embeds) is far more
GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and
is hard to guess. Add an Auto mode (default ON) that finds it:

- _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the
  target: grow while throughput keeps improving and VRAM stays under budget,
  revert a step that doesn't help, back off under memory pressure (VRAM >= 90%),
  then settle and periodically re-probe (the GPU/IO balance shifts over a run).
- A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the
  control UI re-enables it. status() reports `auto`; the dial reflects the
  auto-chosen count (read-only) while Auto is on.
- AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 18:19:15 -04:00
bvandeusen 6cd3153bf4 Merge pull request 'Release: SigLIP 2 default (new installs) + embedder model dropdown' (#157) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m26s
2026-06-30 18:01:01 -04:00
bvandeusen 359bc5a283 feat(ml): default to SigLIP 2 (new installs) + model dropdown, no free-text (#1203)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
- Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in)
  — UPDATE applies ONLY where no image is embedded yet (fresh install), so an
  existing library is NOT silently invalidated; it switches deliberately via the
  dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2.
- GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512
  recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types.
- GpuAgentCard: the two name/version text fields → a single model dropdown;
  Save sets name+version from the picked option (the current model is always
  selectable even if off-list).
- embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1)
  to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the
  ml-worker's persistent HF_HOME.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 16:29:27 -04:00
bvandeusen 5f2853168a Merge pull request 'Release: GPU re-process trigger (apply new crop detectors to the existing library)' (#156) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m22s
Build images / build-agent (push) Successful in 3m35s
2026-06-30 16:13:36 -04:00
bvandeusen 80f8eb4756 feat(gpu): re-process trigger to apply new crop detectors to the existing library (#1202)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m25s
The siglip/ccip backfills skip images that already have current-version regions,
so adding crop detectors only affected NEW images — the back-catalogue would
never be re-cropped. Add a reprocess trigger that resets every done/error job of
a task back to pending, so the agent re-runs the FULL pipeline (figure detection
+ CCIP + concept/panel crops) over the whole library under the current detectors.

- reprocess_gpu_jobs(task='ccip') task + POST /api/gpu/reprocess.
- gpu store reprocess() + GpuAgentCard "Re-process library (re-detect + re-crop)"
  button with a confirm (it's heavy).
- Test: a done job resets to pending (attempts cleared).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 16:09:37 -04:00
bvandeusen 9a979ee808 Merge pull request 'Release: Explore diversification, model-swap, Camie/centroid retirement, tag-API, crop proposers' (#155) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m42s
CI / integration (push) Successful in 3m26s
Build images / build-agent (push) Successful in 10m56s
2026-06-30 16:06:13 -04:00
bvandeusen ce5db5caaf chore(agent): default the anatomy + panel proposer weights to working values
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m25s
booru_yolo yolov11m_aa22.pt (40MB) + mosesb/best-comic-panel-detection::best.pt.
Each self-disables if its download fails, so defaulting them on is safe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 15:28:50 -04:00