bvandeusen
48c8811d69
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
39 lines
1.8 KiB
Python
39 lines
1.8 KiB
Python
"""HeadMetricsSnapshot — a daily per-concept time-series point (#114).
|
|
|
|
The "amount of change over time" reporting the operator asked for: once a day,
|
|
record each concept's auto-applied VOLUME (current head_auto tags), cumulative
|
|
misfires/under-fires, and the head's measured quality. Plotting these rows over
|
|
time shows whether auto-apply is landing better/worse and whether tagging more is
|
|
sharpening a concept — the signal for tuning the precision target + support floor.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class HeadMetricsSnapshot(Base):
|
|
__tablename__ = "head_metrics_snapshot"
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
tag_id: Mapped[int] = mapped_column(
|
|
ForeignKey("tag.id", ondelete="CASCADE"), index=True
|
|
)
|
|
# Denormalized so a snapshot stays readable even if the tag is later renamed.
|
|
name: Mapped[str] = mapped_column(String(255), nullable=False)
|
|
snapshot_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
|
|
)
|
|
# Current count of source='head_auto' applications still standing.
|
|
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
# The head's measured quality at snapshot time (null if no head exists).
|
|
ap: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
recall: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
n_pos: Mapped[int | None] = mapped_column(Integer, nullable=True)
|