Graduated heads can now apply their tag without a human — gated so it's safe:
- FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled,
default OFF) is on AND it has >= head_auto_apply_min_positives (default 30)
clean labels. A precise-looking but under-supported low-N head can't spray tags.
- auto_apply_sweep (heads.py): streams every embedded image in chunks, scores
against the eligible heads (numpy, no sklearn), applies each head's tag where
score >= its auto_apply_threshold and the tag isn't already applied/rejected,
with source='head_auto' (distinguishable + reversible). dry_run counts only.
- HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags
task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled)
+ recovery sweep + retention(20).
- API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled),
GET /api/heads/auto-apply (recent runs + per-concept report). Settings
head_auto_apply_enabled + min_positives via /api/ml/settings.
Tests: sweep applies above threshold, dry-run writes nothing, skips under-
supported + ungraduated heads; API disabled/dry-run/conflict guards.
NEXT (slice 2): the observability the operator asked for — per-concept misfire
(auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a
reporting API to tune. Slice 3: the UI (enable, preview, trends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two cadences for keeping heads in sync with your tagging:
- PASSIVE: a nightly `scheduled_train_heads` beat (skips if a run is already
in flight; creates+commits the run row before dispatching train_heads so the
ml worker always finds it). Folds the day's accepts/rejects + newly-eligible
concepts into the heads without anyone clicking.
- ACTIVE: a "Retrain heads" button in the Explore trail bar — bank the +/-
feedback you just gave while walking content, without a trip to Settings.
Shared logic in a new useHeadTraining composable (trigger + poll + start/finish
toasts), used by the Explore button; reflects an already-running run (incl. the
nightly one) on mount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recompute_centroid + recompute_centroids were the only tasks still using
the process-wide singleton extensions.get_session() under asyncio.run().
The async engine's asyncpg pool is bound to the loop it was created on;
each Celery task runs a fresh asyncio.run() loop, so after the first
invocation the cached engine handed loop-A connections to loop B and raised
"Future attached to a different loop" — every recompute after the first in
a worker process failed (~35ms, fails on first DB await).
Convert both to the established per-task async_session_factory() pattern
(NullPool engine created + disposed inside the task's own loop), matching
scan/download/admin tasks. No get_session usages remain in tasks/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag
firing on one frame survived at peak confidence, and a fixed count under-samples
long multi-scene videos so real scene-local tags looked like noise.
Redesign (operator-steered):
- Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds`
(default 4) across the 5–95% window — so a tag's frame-presence reflects real
screen time independent of video length. Capped at `video_max_frames` (default
64): a long video stretches the spacing instead of exploding into hundreds of
inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg
timeout also cut 60s→30s).
- Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in
>= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on
screen — duration-independent noise rejection), with confidence = MEAN over the
frames it appears in (not max). Clamps the threshold to the sample count so a
1–2-frame short video still tags.
- All three knobs are DB-backed ml_settings (migration 0053), patchable via
/api/ml/settings + sliders in the ML settings card — replaces the
VIDEO_ML_FRAMES env var (product-not-project).
Tests: aggregation drops one-frame noise + means corroborated tags + clamps on
short videos; settings round-trip + min>max validation. Replaced the
_maxpool_predictions unit test.
NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs
CPU-only — is GPU enablement, tracked separately in #872.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch every prediction READER off the JSON column onto the normalized
image_prediction table. Parity by construction: each reader loads the same
{raw_name: {category, confidence}} dict it consumed before (via small
_load_predictions helpers), so all downstream threshold/alias/merge/consensus
logic is byte-identical — only the data source changed.
- suggestions.SuggestionService.for_image (and for_selection via it)
- ml.apply_allowlist_tags (iterates images that have prediction rows)
- importer re-import reset deletes the image's prediction rows
The tagger_predictions JSON column is still dual-written (step 1) so it stays
valid during transition; the backfill task's NULL check still works. Removing
the JSON write + DROP column + retiring the #764 prune is the cleanup
follow-up (needs a quiesced-worker window for the DROP lock).
Tests: shared tests/_prediction_helpers.seed_predictions seeds the table;
read-path tests (suggestions, bulk consensus, allowlist apply, API) seed there
instead of ImageRecord.tagger_predictions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize tagger predictions out of the image_record.tagger_predictions JSON
blob into a queryable per-prediction table. Step 1 of the cutover (expand):
additive + low-risk — reads still use the JSON, this just adds the table and
keeps it populated.
- ImagePrediction(image_record_id, raw_name, category, score) — stores the
RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution
is unchanged. Indexed for per-image reads + by (raw_name, score).
- Migration 0045: create table + set-based backfill from the JSON via
json_each (fast post-#764-prune). The old column stays (vestigial) and is
dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the
hot image_record table, so it waits for a quiesced-worker window.
- tag_and_embed dual-writes the rows (delete-then-insert, idempotent);
tagger_store_floor already applied in infer().
Next: switch suggestion + allowlist reads to the table, then drop the JSON
write. Plan-task #768.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default
0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70).
Storing every tag down to 0.05 from a ~10k-tag tagger is what grew
image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70
and the centroid/learned path covers lower-confidence preferred tags, so the
sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim
land next; this only changes the write gate for NEW imports).
- ml_settings.tagger_store_floor (migration 0044, default 0.70)
- tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor
- ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold
below the floor (nothing below the floor is stored, so the gap surfaces
nothing) — server backstop for the UI slider clamp
- Settings → ML: store-floor slider + caption; category sliders min-bound to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator: 10-frame max-pooled tagging on video produces a lot of noisy tags, and
the sampling burns time/GPU. Drop the VIDEO_ML_FRAMES default to 6 (still env-
overridable). Fewer frames = less per-frame noise into the max-pool and a smaller
frame-sampling budget. Quality/perf of the whole video path is being reviewed
separately.
The task logged nothing and SoftTimeLimitExceeded stringifies to empty, so a
timeout surfaced as a bare 'SoftTimeLimitExceeded()' with no clue which file or
why (operator-flagged 2026-06-08).
- Log start (id/path/mime/bytes/video?), per-phase timing (load_models, video
probe/sample/infer, tag, embed, persist), and a success summary.
- Track a + file ; on SoftTimeLimitExceeded log it and re-raise
SoftTimeLimitExceeded WITH that context (keeps the 'timeout' task_run status
but gives the activity a real error_message: which file, which phase, elapsed).
- On other exceptions, log context then re-raise the ORIGINAL (preserves
autoretry for OSError/DBAPIError/OperationalError).
Now a stuck run names the culprit — most likely a slow video (frame sampling is
up to 10x60s ffmpeg) or a huge image; the phase log will say which.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.
New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):
- probe_video(path): validates the container + first video stream via
ffprobe (a separate binary — a decoder crash kills only ffprobe, not
the worker). Returns width/height, which the importer didn't capture
for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
(MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
spawned child process. A decompression-bomb OOM or native-lib
segfault on a malformed archive shows up as a non-zero child exit
code → crashed=True, never a dead worker.
ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).
Wired:
- importer._import_media video branch: probe_video before the pipeline;
crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
→ failed, clean reject → still preserve the archive as a
PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
so a corrupt video is rejected (status='bad_video') instead of
crashing the ml-worker on frame decode.
Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.
Root cause is two interacting limits, both undersized for video work:
tag_and_embed: soft_time_limit=300, time_limit=420
(sized for the image branch, ≈2 GPU ops)
recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues
The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.
**Two-part fix**
1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
(20 min). Sized for the video path's worst case; image runs return
in seconds and don't care.
2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
Queues with legitimately-long-running tasks (currently just `ml` at
25 min — 5-min buffer past the new hard kill) get their own
threshold; queues not in the dict use the default 5 min. The sweep
now issues one UPDATE per distinct threshold value, with
`queue.notin_(override_queues)` on the default pass so each row is
touched at most once.
Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
pins both directions: a 10-min-old ml row survives (fresh by 25-min
override), a 30-min-old ml row gets flagged.
After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
apply_allowlist_tags: 4 modes (tag-only / image-only / both / full sweep),
matches a tag to a prediction either by direct name or via alias
(name, category) resolution, gates on per-tag min_confidence, skips
applied/rejected, applies source='ml_auto'. recompute_centroid /
recompute_centroids: async-bridged calls into CentroidService, delta-gated.
Beat: daily backfill, daily centroid recompute, daily allowlist sweep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tag_and_embed: Camie + SigLIP on one image (video → 10-frame sample,
max-pool tags, mean-pool embeddings), stores predictions/embedding with
model versions, then enqueues per-image allowlist apply. backfill:
keyset-paginated discovery of images missing predictions/embeddings for
the current model versions (restart-safe). apply_allowlist_tags stub
included so .delay() resolves between commits (filled in Task 9).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen