feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m19s

Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag
firing on one frame survived at peak confidence, and a fixed count under-samples
long multi-scene videos so real scene-local tags looked like noise.

Redesign (operator-steered):
- Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds`
  (default 4) across the 5–95% window — so a tag's frame-presence reflects real
  screen time independent of video length. Capped at `video_max_frames` (default
  64): a long video stretches the spacing instead of exploding into hundreds of
  inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg
  timeout also cut 60s→30s).
- Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in
  >= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on
  screen — duration-independent noise rejection), with confidence = MEAN over the
  frames it appears in (not max). Clamps the threshold to the sample count so a
  1–2-frame short video still tags.
- All three knobs are DB-backed ml_settings (migration 0053), patchable via
  /api/ml/settings + sliders in the ML settings card — replaces the
  VIDEO_ML_FRAMES env var (product-not-project).

Tests: aggregation drops one-frame noise + means corroborated tags + clamps on
short videos; settings round-trip + min>max validation. Replaced the
_maxpool_predictions unit test.

NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs
CPU-only — is GPU enablement, tracked separately in #872.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 11:07:00 -04:00
parent 41652db20f
commit 369e3de684
7 changed files with 241 additions and 35 deletions
+15
View File
@@ -14,6 +14,9 @@ _EDITABLE = (
"centroid_similarity_threshold",
"min_reference_images",
"tagger_store_floor",
"video_frame_interval_seconds",
"video_max_frames",
"video_min_tag_frames",
)
@@ -32,6 +35,9 @@ async def get_settings():
"centroid_similarity_threshold": s.centroid_similarity_threshold,
"min_reference_images": s.min_reference_images,
"tagger_store_floor": s.tagger_store_floor,
"video_frame_interval_seconds": s.video_frame_interval_seconds,
"video_max_frames": s.video_max_frames,
"video_min_tag_frames": s.video_min_tag_frames,
"tagger_model_version": s.tagger_model_version,
"embedder_model_version": s.embedder_model_version,
}
@@ -85,6 +91,15 @@ def _validate(p: dict) -> str | None:
f"suggestion_threshold_{cat} cannot be below tagger_store_floor "
f"({floor}) — predictions below the floor are not stored"
)
# Video tagging (#747).
if p["video_frame_interval_seconds"] <= 0:
return "video_frame_interval_seconds must be > 0"
if p["video_max_frames"] < 1:
return "video_max_frames must be >= 1"
if p["video_min_tag_frames"] < 1:
return "video_min_tag_frames must be >= 1"
if p["video_min_tag_frames"] > p["video_max_frames"]:
return "video_min_tag_frames cannot exceed video_max_frames"
return None