feat(heads): production per-concept heads — train + score backend (#114 A)
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its production form (the first of three slices that make heads the suggestion source, replacing Camie + centroid). - tag_head: one logistic-regression head per general/character concept with enough labelled positives. Weights (pgvector), honest CV-derived suggest threshold + earned-auto-apply point, and per-concept quality metrics. - head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the admin card shows live + historical status across navigation. - services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data loaders + metric math so production heads match measured eval numbers) and SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one image's embedding against all heads → the rail's suggestions, cached on (count, max trained_at) so a retrain invalidates without per-request loads. - tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress) + recover_stalled_head_training_runs sweep + retention(20) + 5-min beat (rule 89). - api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET /api/heads (count, graduated, last-trained, running, per-concept table, recent runs). - ml_settings: head_min_positives + head_auto_apply_precision, tunable via /api/ml/settings. Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B — this slice makes training + scoring exist and CI-verifiable. 'precision' column stored as precision_cv (SQL reserved word). Migration 0058. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -583,3 +583,49 @@ def tag_eval_run(self, run_id: int) -> str:
|
||||
run.finished_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
return "ready"
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.ml.train_heads",
|
||||
bind=True,
|
||||
# Trains a logistic-regression head per eligible concept over stored SigLIP
|
||||
# embeddings — minutes for a full library. Runs on the ml queue (only that
|
||||
# worker has scikit-learn). Commits per head so a kill leaves progress.
|
||||
soft_time_limit=3600, time_limit=3900,
|
||||
)
|
||||
def train_heads(self, run_id: int) -> str:
|
||||
"""(Re)train all eligible concept heads into tag_head, tracked by the
|
||||
HeadTrainingRun row so the admin card shows live + historical status."""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from ..models import HeadTrainingRun
|
||||
from ..services.ml.heads import train_all_heads
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
run = session.get(HeadTrainingRun, run_id)
|
||||
if run is None:
|
||||
return "missing"
|
||||
run.last_progress_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
try:
|
||||
result = train_all_heads(session, run.params, run)
|
||||
except SoftTimeLimitExceeded:
|
||||
run.status = "error"
|
||||
run.error = "timed out"
|
||||
run.finished_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
log.exception("train_heads %d failed", run_id)
|
||||
run.status = "error"
|
||||
run.error = str(exc)
|
||||
run.finished_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
return "error"
|
||||
run.n_trained = result["n_trained"]
|
||||
run.n_skipped = result["n_skipped"]
|
||||
run.status = "ready"
|
||||
run.finished_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
return "ready"
|
||||
|
||||
Reference in New Issue
Block a user