feat(ml): image_prediction table + backfill + dual-write (#768 step 1)
Normalize tagger predictions out of the image_record.tagger_predictions JSON blob into a queryable per-prediction table. Step 1 of the cutover (expand): additive + low-risk — reads still use the JSON, this just adds the table and keeps it populated. - ImagePrediction(image_record_id, raw_name, category, score) — stores the RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution is unchanged. Indexed for per-image reads + by (raw_name, score). - Migration 0045: create table + set-based backfill from the JSON via json_each (fast post-#764-prune). The old column stays (vestigial) and is dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the hot image_record table, so it waits for a quiesced-worker window. - tag_and_embed dual-writes the rows (delete-then-insert, idempotent); tagger_store_floor already applied in infer(). Next: switch suggestion + allowlist reads to the table, then drop the JSON write. Plan-task #768. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+19
-2
@@ -10,11 +10,11 @@ import logging
|
||||
from pathlib import Path
|
||||
|
||||
from celery.exceptions import SoftTimeLimitExceeded
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.exc import DBAPIError, OperationalError
|
||||
|
||||
from ..celery_app import celery
|
||||
from ..models import ImageRecord, MLSettings
|
||||
from ..models import ImagePrediction, ImageRecord, MLSettings
|
||||
from ._sync_engine import sync_session_factory as _sync_session_factory
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -162,6 +162,23 @@ def tag_and_embed(self, image_id: int) -> dict:
|
||||
record.siglip_embedding = embedding.tolist()
|
||||
record.siglip_model_version = settings.embedder_model_version
|
||||
session.add(record)
|
||||
# Write the normalized image_prediction rows (#768). Delete-then-
|
||||
# insert keeps a re-tag idempotent. tagger_store_floor was already
|
||||
# applied in tagger.infer, so preds is the >=floor set. (Transitional
|
||||
# dual-write alongside the JSON column until the read cutover lands.)
|
||||
session.execute(
|
||||
delete(ImagePrediction).where(
|
||||
ImagePrediction.image_record_id == image_id
|
||||
)
|
||||
)
|
||||
session.add_all([
|
||||
ImagePrediction(
|
||||
image_record_id=image_id, raw_name=name,
|
||||
category=p.get("category", "general"),
|
||||
score=float(p.get("confidence", 0.0)),
|
||||
)
|
||||
for name, p in preds.items()
|
||||
])
|
||||
session.commit()
|
||||
except SoftTimeLimitExceeded:
|
||||
log.error(
|
||||
|
||||
Reference in New Issue
Block a user