feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
+26
-11
@@ -11,6 +11,7 @@ orphaned work is re-picked at once rather than waiting out the lease.
|
||||
"""
|
||||
import threading
|
||||
|
||||
import numpy as np
|
||||
import requests
|
||||
|
||||
from . import media, models
|
||||
@@ -193,28 +194,42 @@ class Worker:
|
||||
else:
|
||||
frames = [(None, media.load_image(data))]
|
||||
|
||||
task = job.get("task") or "ccip"
|
||||
embed_version = job.get("embed_version") or DEFAULT_EMBED_VERSION
|
||||
model_name = (
|
||||
self.cfg.embed_model_override
|
||||
or job.get("embed_model_name")
|
||||
or DEFAULT_EMBED_MODEL
|
||||
)
|
||||
|
||||
# 'embed' = WHOLE-IMAGE SigLIP embedding (re-embed the library under a
|
||||
# new model, #1190) → image_record.siglip_embedding. Mean-pool video
|
||||
# frames, matching the server's tag_and_embed. No regions.
|
||||
if task == "embed":
|
||||
embedder = self._ensure_embedder(model_name)
|
||||
vecs = [embedder.embed(frame) for _, frame in frames]
|
||||
if len(vecs) > 1:
|
||||
vec = np.mean(
|
||||
np.asarray(vecs, dtype=np.float32), axis=0
|
||||
).tolist()
|
||||
else:
|
||||
vec = vecs[0]
|
||||
self.client.submit_embedding(job["job_id"], vec, embed_version)
|
||||
self._bump(processed=1)
|
||||
return True
|
||||
|
||||
# task picks what to produce per crop:
|
||||
# 'siglip' (backfill existing images) → concept (SigLIP) regions
|
||||
# ONLY, so it never churns their figure/CCIP regions or the
|
||||
# character-reference cache.
|
||||
# 'ccip' / 'both' (a new image's first pass) → figure (CCIP) AND
|
||||
# concept (SigLIP) in one go, off the same crop.
|
||||
task = job.get("task") or "ccip"
|
||||
want_ccip = task in ("ccip", "both")
|
||||
want_siglip = task in ("ccip", "siglip", "both")
|
||||
replace_kinds = (
|
||||
["concept"] if task == "siglip" else ["figure", "face", "concept"]
|
||||
)
|
||||
|
||||
embed_version = job.get("embed_version") or DEFAULT_EMBED_VERSION
|
||||
embedder = None
|
||||
if want_siglip:
|
||||
model_name = (
|
||||
self.cfg.embed_model_override
|
||||
or job.get("embed_model_name")
|
||||
or DEFAULT_EMBED_MODEL
|
||||
)
|
||||
embedder = self._ensure_embedder(model_name)
|
||||
embedder = self._ensure_embedder(model_name) if want_siglip else None
|
||||
|
||||
regions = []
|
||||
ccip_ev = self.cfg.ccip_model or "ccip-default"
|
||||
|
||||
Reference in New Issue
Block a user