feat(heads): auto-apply observability + on by default (#114 auto-apply B)
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -846,6 +846,79 @@ def recover_stalled_head_auto_apply_runs() -> int:
|
||||
return recovered
|
||||
|
||||
|
||||
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
|
||||
HEAD_METRICS_SNAPSHOT_RETENTION_DAYS = 180
|
||||
|
||||
|
||||
@celery.task(name="backend.app.tasks.maintenance.snapshot_head_metrics")
|
||||
def snapshot_head_metrics() -> int:
|
||||
"""Daily per-concept observability point (#114): record each head-bearing
|
||||
concept's auto-applied volume, cumulative misfires/under-fires, and the
|
||||
head's measured quality — the time-series the operator tunes from. Prunes
|
||||
points older than the retention window."""
|
||||
from ..models import (
|
||||
HeadMetric,
|
||||
HeadMetricsSnapshot,
|
||||
Tag,
|
||||
TagHead,
|
||||
)
|
||||
from ..models.tag import image_tag
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
now = datetime.now(UTC)
|
||||
with SessionLocal() as session:
|
||||
heads = {
|
||||
r.tag_id: r for r in session.execute(
|
||||
select(
|
||||
TagHead.tag_id, TagHead.ap, TagHead.precision_cv,
|
||||
TagHead.recall, TagHead.n_pos,
|
||||
)
|
||||
)
|
||||
}
|
||||
metrics = {
|
||||
r.tag_id: r for r in session.execute(
|
||||
select(
|
||||
HeadMetric.tag_id, HeadMetric.n_misfires, HeadMetric.n_underfires
|
||||
)
|
||||
)
|
||||
}
|
||||
applied = dict(
|
||||
session.execute(
|
||||
select(image_tag.c.tag_id, func.count())
|
||||
.where(image_tag.c.source == "head_auto")
|
||||
.group_by(image_tag.c.tag_id)
|
||||
)
|
||||
)
|
||||
tag_ids = set(heads) | set(metrics)
|
||||
if not tag_ids:
|
||||
return 0
|
||||
names = dict(
|
||||
session.execute(select(Tag.id, Tag.name).where(Tag.id.in_(tag_ids)))
|
||||
)
|
||||
for tid in tag_ids:
|
||||
h = heads.get(tid)
|
||||
m = metrics.get(tid)
|
||||
session.add(HeadMetricsSnapshot(
|
||||
tag_id=tid, name=names.get(tid, str(tid)),
|
||||
snapshot_at=now,
|
||||
n_auto_applied=applied.get(tid, 0),
|
||||
n_misfires=m.n_misfires if m else 0,
|
||||
n_underfires=m.n_underfires if m else 0,
|
||||
ap=h.ap if h else None,
|
||||
precision_cv=h.precision_cv if h else None,
|
||||
recall=h.recall if h else None,
|
||||
n_pos=h.n_pos if h else None,
|
||||
))
|
||||
session.execute(
|
||||
delete(HeadMetricsSnapshot).where(
|
||||
HeadMetricsSnapshot.snapshot_at
|
||||
< now - timedelta(days=HEAD_METRICS_SNAPSHOT_RETENTION_DAYS)
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return len(tag_ids)
|
||||
|
||||
|
||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
|
||||
def recover_stalled_import_batches() -> int:
|
||||
"""Finalize ImportBatch rows stuck in running past the hard limit
|
||||
|
||||
Reference in New Issue
Block a user