feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)

Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.

- settings: embedder_model_name is now a setting (migration 0065) alongside the
  existing embedder_model_version; both editable + validated (non-empty) in the
  ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
  model-agnostic), preferring the pre-downloaded local dir for the default so
  existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
  under the lease-announced model → POST /jobs/submit_embedding writes
  image_record.siglip_embedding + siglip_model_version. The lease now announces
  the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
  version images; 'siglip' now re-embeds concept crops whose version != current
  (so a swap re-triggers crops, not just the never-embedded back-catalogue). The
  CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
  churn the library at 512px) — the GPU agent owns version re-embeds. Daily
  'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
  image gated by siglip_model_version, concept regions by embedding_version) so a
  mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
  "Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.

Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

backend/app/tasks/ml.py

@@ -95,7 +95,7 @@ def tag_and_embed(self, image_id: int) -> dict:
 
             phase = "load_models"
             tagger = get_tagger()
-            embedder = get_embedder()
+            embedder = get_embedder(settings.embedder_model_name)
 
             if is_vid:
                 # Layer-3 isolation: ffprobe (a separate process) validates
@@ -330,10 +330,10 @@ def backfill(self) -> int:
                         != settings.tagger_model_version
                     )
                     | (ImageRecord.siglip_embedding.is_(None))
-                    | (
-                        ImageRecord.siglip_model_version
-                        != settings.embedder_model_version
-                    )
+                    # NB: a siglip MODEL-VERSION mismatch (an operator model swap,
+                    # #1190) is intentionally NOT re-embedded here — the CPU
+                    # ml-worker can't churn the whole library at 384/512px. The
+                    # GPU agent owns version re-embeds via the 'embed' job.
                 )
                 .order_by(ImageRecord.id.asc())
                 .limit(500)
@@ -750,17 +750,40 @@ def enqueue_gpu_backfill(task_name: str) -> int:
     job, so it picks up the back-catalogue of images that were CCIP-embedded
     before concept crops existed, and retries images whose concept embed failed —
     without re-touching their figure/CCIP regions."""
-    from sqlalchemy import exists, insert, literal
+    from sqlalchemy import exists, insert, literal, or_
     from sqlalchemy import select as sa_select
 
-    from ..models import GpuJob, ImageRecord, ImageRegion
+    from ..models import GpuJob, ImageRecord, ImageRegion, MLSettings
 
     SessionLocal = _sync_session_factory()
     with SessionLocal() as session:
-        if task_name == "siglip":
-            has_concept = exists().where(
+        cur_version = session.execute(
+            select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
+        ).scalar_one()
+        if task_name == "embed":
+            # Whole-image GPU re-embed (#1190): images with no embedding, or one
+            # stamped under a DIFFERENT model version (an operator model swap).
+            stale = or_(
+                ImageRecord.siglip_embedding.is_(None),
+                ImageRecord.siglip_model_version.is_(None),
+                ImageRecord.siglip_model_version != cur_version,
+            )
+            queued = exists().where(
+                GpuJob.image_record_id == ImageRecord.id,
+                GpuJob.task == "embed",
+                GpuJob.status.in_(["pending", "leased"]),
+            )
+            sel = sa_select(
+                ImageRecord.id, literal("embed"), literal("pending")
+            ).where(stale).where(~queued)
+        elif task_name == "siglip":
+            # Concept-crop re-embed: enqueue when there's no concept region AT THE
+            # CURRENT model version — so a model swap re-triggers crops too, not
+            # only the never-embedded back-catalogue.
+            has_current_concept = exists().where(
                 ImageRegion.image_record_id == ImageRecord.id,
                 ImageRegion.kind == "concept",
+                ImageRegion.embedding_version == cur_version,
             )
             queued = exists().where(
                 GpuJob.image_record_id == ImageRecord.id,
@@ -769,7 +792,7 @@ def enqueue_gpu_backfill(task_name: str) -> int:
             )
             sel = sa_select(
                 ImageRecord.id, literal("siglip"), literal("pending")
-            ).where(~has_concept).where(~queued)
+            ).where(~has_current_concept).where(~queued)
         else:
             already = exists().where(
                 GpuJob.image_record_id == ImageRecord.id,