feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
+34
-6
@@ -17,7 +17,6 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from ..extensions import get_session
|
||||
from ..models import AppSetting, GpuJob, ImageRecord, MLSettings
|
||||
from ..services.gallery_service import image_url
|
||||
from ..services.ml.embedder import MODEL_NAME as EMBED_MODEL_NAME
|
||||
from ..services.ml.gpu_jobs import GpuJobService
|
||||
from ..services.ml.regions import RegionService
|
||||
|
||||
@@ -138,11 +137,12 @@ async def lease():
|
||||
# For video/animated: the agent samples at this cadence.
|
||||
"frame_interval_seconds": ml.video_frame_interval_seconds,
|
||||
"max_frames": ml.video_max_frames,
|
||||
# The embedding model the agent must use for concept crops, so
|
||||
# its region vectors land in the SAME space the heads trained in.
|
||||
# Server-announced → the agent stays model-agnostic; a swap is a
|
||||
# server setting + a re-embed migration, never an agent change.
|
||||
"embed_model_name": EMBED_MODEL_NAME,
|
||||
# The embedding model the agent must use for concept crops + the
|
||||
# whole-image 'embed' task, so its vectors land in the SAME space
|
||||
# the heads trained in. Server-announced FROM THE SETTING → the
|
||||
# agent stays model-agnostic; an operator swap is a setting + a
|
||||
# re-embed, never an agent change.
|
||||
"embed_model_name": ml.embedder_model_name,
|
||||
"embed_version": ml.embedder_model_version,
|
||||
})
|
||||
return jsonify({"jobs": out})
|
||||
@@ -188,6 +188,34 @@ async def submit():
|
||||
return jsonify({"ok": True, "stored": len(regions)})
|
||||
|
||||
|
||||
@gpu_bp.route("/jobs/submit_embedding", methods=["POST"])
|
||||
async def submit_embedding():
|
||||
"""Store a whole-image SigLIP embedding (the 'embed' task) on image_record +
|
||||
close the job. Body: {agent_id, job_id, embedding:[...], embedding_version}.
|
||||
This is how the GPU agent re-embeds the library under a new model (#1190) —
|
||||
much faster than the CPU ml-worker at higher resolutions."""
|
||||
body = await request.get_json(silent=True) or {}
|
||||
agent_id = str(body.get("agent_id") or "agent")
|
||||
job_id = body.get("job_id")
|
||||
embedding = body.get("embedding")
|
||||
version = body.get("embedding_version")
|
||||
if job_id is None or not embedding or not version:
|
||||
return jsonify({"error": "job_id, embedding, embedding_version required"}), 400
|
||||
async with get_session() as session:
|
||||
if not await _agent_authed(session):
|
||||
return jsonify({"error": "unauthorized"}), 401
|
||||
job = await session.get(GpuJob, int(job_id))
|
||||
if job is None or job.status != "leased" or job.lease_token != agent_id:
|
||||
return jsonify({"error": "lease_invalid"}), 409
|
||||
img = await session.get(ImageRecord, job.image_record_id)
|
||||
if img is not None:
|
||||
img.siglip_embedding = embedding
|
||||
img.siglip_model_version = version
|
||||
await GpuJobService(session).complete(agent_id, int(job_id))
|
||||
await session.commit()
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@gpu_bp.route("/jobs/fail", methods=["POST"])
|
||||
async def fail():
|
||||
body = await request.get_json(silent=True) or {}
|
||||
|
||||
Reference in New Issue
Block a user