feat(tagging): SigLIP concept crops + max-over-bag scoring (#114) #153

2026-06-30T08:18:17-04:00

bvandeusen commented

2026-06-30 08:18:17 -04:00

What

Lifts recall on small/local concepts (glasses, cum, stomach-bulge, xray, lactation, …) that the whole-image SigLIP vector washes out. The GPU agent now embeds figure crops with SigLIP too (stored as kind='concept' regions), and the suggestion rail scores each image as a bag — whole-image embedding + every concept crop — taking each head's max over the bag. The whole-image vector is always in the bag, so a score can never come out lower than today.

Model-agnostic by construction

The server announces the embedding model (HF name + version) in the lease, so the agent loads whatever model the heads were trained in and stays in lock-step. A model swap is a server setting + a re-embed migration — never an agent change. (Caveat: pgvector columns are fixed-width, so a different output dim needs a column-resize migration; folds into the re-embed.)

Changes

Agent (outside CI; built by build-agent job)

embedder.py — model-agnostic CropEmbedder (torch/transformers get_image_features, fp16 on CUDA, inference-locked single shared instance).
worker.py — branches on job.task: ccip/both emit figure(CCIP)+concept(SigLIP) in one pass; siglip emits concept-only so the back-catalogue backfill never churns figure/CCIP regions or the character-reference cache.
requirements.txt + Dockerfile — torch from the cu124 wheel index + transformers.

Server

api/gpu.py — lease announces embed_model_name / embed_version.
heads.score_image — max-over-bag (version-filtered region embeddings).
tasks/ml.enqueue_gpu_backfill — siglip branch gates on a missing concept region (drains the back-catalogue, retries failed embeds, no double-enqueue).
celery_app — daily siglip backfill beat.
api/ccip.overview — reports images_with_concept_siglip.
Frontend — "Queue concept crops (SigLIP)" button.

Scope (v1)

Suggestion rail only. Auto-apply stays whole-image (conservative — heads' thresholds were calibrated on whole-image; max-over-bag can inflate scores, and rail extra-recall is operator-reviewed). Bulk-apply on bags is a documented follow-up.

Tests

test_ml_suggestions::test_concept_region_surfaces_via_max_over_bag — a crop aligned with a head lifts a sub-threshold whole-image score over the cut; a stale-version region is filtered out of the bag.
test_gpu_jobs::test_enqueue_siglip_backfill_gates_on_concept_region — selection + idempotency.

🤖 Generated with Claude Code

## What Lifts recall on **small/local concepts** (glasses, cum, stomach-bulge, xray, lactation, …) that the whole-image SigLIP vector washes out. The GPU agent now embeds figure crops with SigLIP too (stored as `kind='concept'` regions), and the suggestion rail scores each image as a **bag** — whole-image embedding + every concept crop — taking each head's **max over the bag**. The whole-image vector is always in the bag, so a score can never come out *lower* than today. ## Model-agnostic by construction The server **announces the embedding model** (HF name + version) in the lease, so the agent loads whatever model the heads were trained in and stays in lock-step. A model swap is a server setting + a re-embed migration — never an agent change. (Caveat: pgvector columns are fixed-width, so a different output dim needs a column-resize migration; folds into the re-embed.) ## Changes **Agent** (outside CI; built by build-agent job) - `embedder.py` — model-agnostic `CropEmbedder` (torch/transformers `get_image_features`, fp16 on CUDA, inference-locked single shared instance). - `worker.py` — branches on `job.task`: `ccip`/`both` emit figure(CCIP)+concept(SigLIP) in one pass; `siglip` emits **concept-only** so the back-catalogue backfill never churns figure/CCIP regions or the character-reference cache. - `requirements.txt` + `Dockerfile` — torch from the cu124 wheel index + transformers. **Server** - `api/gpu.py` — lease announces `embed_model_name` / `embed_version`. - `heads.score_image` — **max-over-bag** (version-filtered region embeddings). - `tasks/ml.enqueue_gpu_backfill` — `siglip` branch gates on a **missing concept region** (drains the back-catalogue, retries failed embeds, no double-enqueue). - `celery_app` — daily `siglip` backfill beat. - `api/ccip.overview` — reports `images_with_concept_siglip`. - Frontend — "Queue concept crops (SigLIP)" button. ## Scope (v1) Suggestion **rail only**. Auto-apply stays whole-image (conservative — heads' thresholds were calibrated on whole-image; max-over-bag can inflate scores, and rail extra-recall is operator-reviewed). Bulk-apply on bags is a documented follow-up. ## Tests - `test_ml_suggestions::test_concept_region_surfaces_via_max_over_bag` — a crop aligned with a head lifts a sub-threshold whole-image score over the cut; a stale-version region is filtered out of the bag. - `test_gpu_jobs::test_enqueue_siglip_backfill_gates_on_concept_region` — selection + idempotency. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 1 commit 2026-06-30 08:18:17 -04:00

feat(tagging): SigLIP concept crops + max-over-bag scoring (#114 ) c6f38b0dac

Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray,
lactation) that the whole-image SigLIP vector washes out: the GPU agent now
embeds figure crops with SigLIP too, stored as kind='concept' regions, and the
suggestion rail scores each image as a BAG (whole-image + every concept crop),
taking each head's MAX over the bag. The whole-image vector is always in the
bag, so this can never score lower than before.

Model-agnostic by construction: the server ANNOUNCES the embedding model
(HF name + version) in the lease, so the agent loads whatever the heads were
trained in and stays in lock-step — a model swap is a server setting + a
re-embed migration, never an agent change.

- agent: model-agnostic CropEmbedder (torch/transformers get_image_features,
  fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits
  figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the
  back-catalogue backfill never churns figure/CCIP regions; torch cu124 +
  transformers in the image.
- server: lease announces embed_model_name/embed_version; score_image is
  max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill
  'siglip' gates on a missing concept region (drains the back-catalogue,
  retries failures, no double-enqueue); daily siglip-backfill beat; UI button;
  /api/ccip/overview reports images_with_concept_siglip.
- v1 scope: suggestion rail only — auto-apply stays whole-image (conservative;
  heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

bvandeusen added 1 commit 2026-06-30 08:34:07 -04:00

feat(agent): survive + auto-recover when curator is unreachable 55fa4656ff

For redeploying curator while away with nobody to restart the agent:

- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
  401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
  fault. On a transport error it hands the job back (best effort) and signals
  the loop to back off — instead of calling fail(), which would burn the job's
  server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
  across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
  re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
  resets on the first successful lease, so a long outage is gentle and recovery
  is automatic within ≤60s of curator returning. Sleeps are interruptible so
  Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
  so a host reboot / crash-restart (restart: unless-stopped) self-heals with
  nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
  work" banner so the recovering state reads as recovery, not failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

bvandeusen merged commit 715e276c03 into dev

2026-06-30 08:40:24 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#153