feat(heads): auto-apply observability + on by default (#114 auto-apply B)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Failing after 3m25s

Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.

Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
  UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
  Both captured at correction time in TagService.add_to_image/remove_from_image
  (source is lost on delete) into durable per-tag counters (head_metric), keyed
  by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
  (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
  + head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
  head quality, plus the snapshot time-series — the report to tune the precision
  target + support floor.

Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.

What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.

NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-29 00:36:58 -04:00
parent 01933c5b26
commit 48c8811d69
11 changed files with 493 additions and 7 deletions
+52 -1
View File
@@ -9,7 +9,7 @@ from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Tag, TagKind, image_tag
from ..models import HeadMetric, Tag, TagHead, TagKind, image_tag
from ..models.tag_allowlist import TagAllowlist
from ..models.tag_reference_embedding import TagReferenceEmbedding
from .db_helpers import get_or_create
@@ -215,6 +215,18 @@ class TagService:
async def add_to_image(self, image_id: int, tag_id: int, source: str = "manual") -> None:
"""Idempotent: re-adding an existing tag does nothing."""
# A genuinely-new MANUAL add of a tag that already has a head is an
# UNDER-FIRE signal — the auto-system should have caught it (#114 obs).
is_new = source == "manual" and (
await self.session.execute(
select(image_tag.c.tag_id).where(
and_(
image_tag.c.image_record_id == image_id,
image_tag.c.tag_id == tag_id,
)
)
)
).first() is None
stmt = pg_insert(image_tag).values(
image_record_id=image_id, tag_id=tag_id, source=source
)
@@ -222,8 +234,22 @@ class TagService:
index_elements=["image_record_id", "tag_id"]
)
await self.session.execute(stmt)
if is_new:
await self._note_under_fire(tag_id)
async def remove_from_image(self, image_id: int, tag_id: int) -> None:
# Removing an auto-applied (source='head_auto') tag is a MISFIRE — read
# the source BEFORE deleting, since it's lost with the row (#114 obs).
src = (
await self.session.execute(
select(image_tag.c.source).where(
and_(
image_tag.c.image_record_id == image_id,
image_tag.c.tag_id == tag_id,
)
)
)
).scalar_one_or_none()
await self.session.execute(
image_tag.delete().where(
and_(
@@ -232,6 +258,31 @@ class TagService:
)
)
)
if src == "head_auto":
await self._bump_metric(tag_id, "n_misfires")
async def _note_under_fire(self, tag_id: int) -> None:
"""Count an under-fire only when the tag actually has a head."""
has_head = (
await self.session.execute(
select(TagHead.tag_id).where(TagHead.tag_id == tag_id)
)
).first() is not None
if has_head:
await self._bump_metric(tag_id, "n_underfires")
async def _bump_metric(self, tag_id: int, column: str) -> None:
"""Increment a HeadMetric counter (upsert), keyed by tag so it survives
head retrain/prune."""
col = HeadMetric.__table__.c[column]
await self.session.execute(
pg_insert(HeadMetric)
.values(tag_id=tag_id, **{column: 1})
.on_conflict_do_update(
index_elements=["tag_id"],
set_={column: col + 1, "updated_at": func.now()},
)
)
async def list_for_image(self, image_id: int) -> Sequence:
"""Tags on an image, ordered (kind, name). Each row carries the fandom's