feat(heads): auto-apply observability + on by default (#114 auto-apply B)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Failing after 3m25s

Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.

Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
  UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
  Both captured at correction time in TagService.add_to_image/remove_from_image
  (source is lost on delete) into durable per-tag counters (head_metric), keyed
  by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
  (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
  + head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
  head quality, plus the snapshot time-series — the report to tune the precision
  target + support floor.

Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.

What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.

NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-29 00:36:58 -04:00
parent 01933c5b26
commit 48c8811d69
11 changed files with 493 additions and 7 deletions
+4
View File
@@ -9,6 +9,8 @@ from .credential import Credential
from .download_event import DownloadEvent
from .external_link import ExternalLink
from .head_auto_apply_run import HeadAutoApplyRun
from .head_metric import HeadMetric
from .head_metrics_snapshot import HeadMetricsSnapshot
from .head_training_run import HeadTrainingRun
from .image_prediction import ImagePrediction
from .image_provenance import ImageProvenance
@@ -69,6 +71,8 @@ __all__ = [
"LibraryAuditRun",
"MLSettings",
"HeadAutoApplyRun",
"HeadMetric",
"HeadMetricsSnapshot",
"HeadTrainingRun",
"TagAlias",
"TagAllowlist",
+32
View File
@@ -0,0 +1,32 @@
"""HeadMetric — running correction counters per concept (#114 observability).
Earned auto-apply fires graduated heads; to TUNE it we need to know how often a
head's auto-applied tag was wrong (the operator removed it = a MISFIRE) and how
often the operator had to add a tag a head exists for by hand (an UNDER-FIRE,
the head missed it). image_tag.source is lost when a row is deleted, so these
are captured as durable cumulative counters at correction time — they survive
head retrain/prune (keyed by tag, not by the head row). The daily snapshot reads
them into the time-series.
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadMetric(Base):
__tablename__ = "head_metric"
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
)
# An auto-applied (source='head_auto') tag the operator later REMOVED.
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# A tag with a head that the operator added by HAND (the head missed it).
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
@@ -0,0 +1,38 @@
"""HeadMetricsSnapshot — a daily per-concept time-series point (#114).
The "amount of change over time" reporting the operator asked for: once a day,
record each concept's auto-applied VOLUME (current head_auto tags), cumulative
misfires/under-fires, and the head's measured quality. Plotting these rows over
time shows whether auto-apply is landing better/worse and whether tagging more is
sharpening a concept — the signal for tuning the precision target + support floor.
"""
from datetime import datetime
from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadMetricsSnapshot(Base):
__tablename__ = "head_metrics_snapshot"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), index=True
)
# Denormalized so a snapshot stays readable even if the tag is later renamed.
name: Mapped[str] = mapped_column(String(255), nullable=False)
snapshot_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
)
# Current count of source='head_auto' applications still standing.
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# The head's measured quality at snapshot time (null if no head exists).
ap: Mapped[float | None] = mapped_column(Float, nullable=True)
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
recall: Mapped[float | None] = mapped_column(Float, nullable=True)
n_pos: Mapped[int | None] = mapped_column(Integer, nullable=True)
+5 -4
View File
@@ -75,12 +75,13 @@ class MLSettings(Base):
Float, nullable=False, default=0.97
)
# Earned auto-apply (#114). A graduated head fires (tags images without a
# human) ONLY when this master switch is on AND the head has at least
# human) when this master switch is on AND the head has at least
# head_auto_apply_min_positives clean labels — so a precise-looking but
# under-supported low-N head can't spray tags across the library. Off by
# default; the operator enables after previewing. Operator-tunable.
# under-supported low-N head can't spray tags across the library. ON by
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
# measured-precision gates keep it safe, and every auto-tag is reversible.
head_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False
Boolean, nullable=False, default=True
)
head_auto_apply_min_positives: Mapped[int] = mapped_column(
Integer, nullable=False, default=30