feat(heads): auto-apply observability + on by default (#114 auto-apply B)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Failing after 3m25s

Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.

Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
  UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
  Both captured at correction time in TagService.add_to_image/remove_from_image
  (source is lost on delete) into durable per-tag counters (head_metric), keyed
  by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
  (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
  + head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
  head quality, plus the snapshot time-series — the report to tune the precision
  target + support floor.

Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.

What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.

NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-29 00:36:58 -04:00
parent 01933c5b26
commit 48c8811d69
11 changed files with 493 additions and 7 deletions
+73
View File
@@ -846,6 +846,79 @@ def recover_stalled_head_auto_apply_runs() -> int:
return recovered
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
HEAD_METRICS_SNAPSHOT_RETENTION_DAYS = 180
@celery.task(name="backend.app.tasks.maintenance.snapshot_head_metrics")
def snapshot_head_metrics() -> int:
"""Daily per-concept observability point (#114): record each head-bearing
concept's auto-applied volume, cumulative misfires/under-fires, and the
head's measured quality — the time-series the operator tunes from. Prunes
points older than the retention window."""
from ..models import (
HeadMetric,
HeadMetricsSnapshot,
Tag,
TagHead,
)
from ..models.tag import image_tag
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
heads = {
r.tag_id: r for r in session.execute(
select(
TagHead.tag_id, TagHead.ap, TagHead.precision_cv,
TagHead.recall, TagHead.n_pos,
)
)
}
metrics = {
r.tag_id: r for r in session.execute(
select(
HeadMetric.tag_id, HeadMetric.n_misfires, HeadMetric.n_underfires
)
)
}
applied = dict(
session.execute(
select(image_tag.c.tag_id, func.count())
.where(image_tag.c.source == "head_auto")
.group_by(image_tag.c.tag_id)
)
)
tag_ids = set(heads) | set(metrics)
if not tag_ids:
return 0
names = dict(
session.execute(select(Tag.id, Tag.name).where(Tag.id.in_(tag_ids)))
)
for tid in tag_ids:
h = heads.get(tid)
m = metrics.get(tid)
session.add(HeadMetricsSnapshot(
tag_id=tid, name=names.get(tid, str(tid)),
snapshot_at=now,
n_auto_applied=applied.get(tid, 0),
n_misfires=m.n_misfires if m else 0,
n_underfires=m.n_underfires if m else 0,
ap=h.ap if h else None,
precision_cv=h.precision_cv if h else None,
recall=h.recall if h else None,
n_pos=h.n_pos if h else None,
))
session.execute(
delete(HeadMetricsSnapshot).where(
HeadMetricsSnapshot.snapshot_at
< now - timedelta(days=HEAD_METRICS_SNAPSHOT_RETENTION_DAYS)
)
)
session.commit()
return len(tag_ids)
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
def recover_stalled_import_batches() -> int:
"""Finalize ImportBatch rows stuck in running past the hard limit