bvandeusen
19b962f1a7
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
97 lines
4.5 KiB
Python
97 lines
4.5 KiB
Python
"""MLSettings — single-row table holding ML pipeline tunables."""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
Boolean,
|
|
CheckConstraint,
|
|
DateTime,
|
|
Float,
|
|
Integer,
|
|
String,
|
|
func,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class MLSettings(Base):
|
|
__tablename__ = "ml_settings"
|
|
# Bare name — Base.metadata's naming convention prepends ck_<table>_,
|
|
# producing the final ck_ml_settings_singleton (matches migration 0003).
|
|
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
# CPU whole-image embedding (B3, operator 2026-07-02). The ml-worker's ONLY
|
|
# processing role is the embed fallback for stacks WITHOUT a GPU agent — ON
|
|
# by default so a fresh install works with no agent. Stacks that run the
|
|
# agent and drop the ml-worker container turn this OFF so import hooks stop
|
|
# queueing embed work nothing will consume (the daily GPU 'embed' backfill
|
|
# covers those images instead).
|
|
cpu_embed_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True
|
|
)
|
|
# Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not
|
|
# a fixed count) so coverage reflects real screen time regardless of length;
|
|
# cap the total so a long video can't explode into hundreds of embeds. The
|
|
# per-frame SigLIP embeddings are mean-pooled. Operator-tunable.
|
|
video_frame_interval_seconds: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=4.0
|
|
)
|
|
video_max_frames: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=64
|
|
)
|
|
# Tagging-v2 head training (#114). The head is the suggestion source that
|
|
# LEARNS from the operator's tags (replacing Camie + centroid). A concept
|
|
# needs >= head_min_positives labelled images before a head is trained;
|
|
# head_auto_apply_precision is the precision bar a head must clear (at some
|
|
# operating point) to "graduate" into earned auto-apply. Operator-tunable.
|
|
head_min_positives: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=8
|
|
)
|
|
head_auto_apply_precision: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=0.97
|
|
)
|
|
# Earned auto-apply (#114). A graduated head fires (tags images without a
|
|
# human) when this master switch is on AND the head has at least
|
|
# head_auto_apply_min_positives clean labels — so a precise-looking but
|
|
# under-supported low-N head can't spray tags across the library. ON by
|
|
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
|
|
# measured-precision gates keep it safe, and every auto-tag is reversible.
|
|
head_auto_apply_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True
|
|
)
|
|
head_auto_apply_min_positives: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=30
|
|
)
|
|
# CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
|
|
# over-fired (high-reference characters matched a scatter of images); 0.85
|
|
# keeps the confident single-character matches. Tunable from the agent card.
|
|
ccip_match_threshold: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=0.85
|
|
)
|
|
# CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
|
|
# above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
|
|
# single-character references + the high bar keep it safe, every tag reversible.
|
|
ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True
|
|
)
|
|
ccip_auto_apply_threshold: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=0.92
|
|
)
|
|
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
|
# existing libraries keep their stored value until the operator re-embeds.
|
|
embedder_model_version: Mapped[str] = mapped_column(
|
|
String(128), nullable=False, default="siglip2-so400m-patch16-512"
|
|
)
|
|
# The HF model NAME the embedder loads (server CPU embed + announced to the
|
|
# GPU agent in the lease). Operator-settable so the embedder is a choice, not
|
|
# a hardcode (#1190): set name + version together, then re-embed + retrain.
|
|
embedder_model_name: Mapped[str] = mapped_column(
|
|
String(128), nullable=False, default="google/siglip2-so400m-patch16-512"
|
|
)
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|