feat(ccip): automation + reference quality — keep identity flowing hands-free (#114)
Works through the optional CCIP ideas + the "keep moving even if I forget" ask:
AUTOMATION (no button needed):
- Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored
ones retried) on their own; the queue never goes idle waiting for a click.
- CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so
identity tags keep flowing. ON by default (opt-out, like head auto-apply);
ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut),
migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips
already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH
/api/ml/settings; auto_applied count in /api/ccip/overview.
REFERENCE QUALITY (the over-fire root cause):
- character_references now draws ONLY from single-character images — on a
multi-character image the tag is image-level, so every figure would otherwise
pollute each character's prototypes (a 2-char image tagged 'Velma' made
Daphne's figure a Velma reference). This is the contamination behind residual
over-firing.
- Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the
reference load isn't redone on every modal open.
Tests: multi-character image not used as a reference; auto-apply tags a confident
match as ccip_auto.
NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot
interesting content").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -13,7 +13,7 @@ exact CCIP difference metric/threshold gets validated against the model during
|
||||
the hands-on eval. numpy is imported lazily (API worker has it via pgvector).
|
||||
"""
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from ...models import ImageRegion, MLSettings, Tag, TagKind
|
||||
@@ -41,10 +41,54 @@ def _l2norm(mat, np):
|
||||
return mat / n
|
||||
|
||||
|
||||
# Single-shot cache of the (expensive) reference load, keyed on a cheap
|
||||
# signature that changes exactly when references could: a character tag added/
|
||||
# removed (n_char_tags) or a figure embedded (max/ n of ccip regions). Shared by
|
||||
# the live matcher (every modal open) and the auto-apply sweep.
|
||||
_REF_CACHE: dict = {"sig": None, "refs": None}
|
||||
|
||||
|
||||
def _single_character_images():
|
||||
"""Subquery of image ids carrying EXACTLY ONE character tag. References come
|
||||
only from these — on a multi-character image the tag is image-level, so every
|
||||
figure would otherwise pollute each character's prototype set (a 2-character
|
||||
image tagged 'Velma' would make Daphne's figure a Velma reference)."""
|
||||
return (
|
||||
select(image_tag.c.image_record_id)
|
||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||
.where(Tag.kind == TagKind.character)
|
||||
.group_by(image_tag.c.image_record_id)
|
||||
.having(func.count() == 1)
|
||||
)
|
||||
|
||||
|
||||
async def _ref_signature(session: AsyncSession) -> tuple:
|
||||
n_tags = (
|
||||
await session.execute(
|
||||
select(func.count())
|
||||
.select_from(image_tag)
|
||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||
.where(Tag.kind == TagKind.character)
|
||||
)
|
||||
).scalar_one()
|
||||
n_regs, max_id = (
|
||||
await session.execute(
|
||||
select(func.count(), func.max(ImageRegion.id)).where(
|
||||
ImageRegion.kind.in_(_FIGURE_KINDS),
|
||||
ImageRegion.ccip_embedding.is_not(None),
|
||||
)
|
||||
)
|
||||
).one()
|
||||
return (n_tags, n_regs, max_id)
|
||||
|
||||
|
||||
async def character_references(session: AsyncSession) -> dict[int, list]:
|
||||
"""Per character-tag CCIP reference vectors: figure/face-region CCIP
|
||||
embeddings on images that carry that character tag (the operator's examples).
|
||||
Multi-prototype — several vectors per character."""
|
||||
embeddings on UNAMBIGUOUS (single-character) images carrying that tag.
|
||||
Multi-prototype — several vectors per character. Cached on a cheap signature."""
|
||||
sig = await _ref_signature(session)
|
||||
if _REF_CACHE["sig"] == sig and _REF_CACHE["refs"] is not None:
|
||||
return _REF_CACHE["refs"]
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(image_tag.c.tag_id, ImageRegion.ccip_embedding)
|
||||
@@ -57,11 +101,13 @@ async def character_references(session: AsyncSession) -> dict[int, list]:
|
||||
.where(Tag.kind == TagKind.character)
|
||||
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
|
||||
.where(ImageRegion.ccip_embedding.is_not(None))
|
||||
.where(ImageRegion.image_record_id.in_(_single_character_images()))
|
||||
)
|
||||
).all()
|
||||
refs: dict[int, list] = {}
|
||||
for tag_id, vec in rows:
|
||||
refs.setdefault(tag_id, []).append(vec)
|
||||
_REF_CACHE.update(sig=sig, refs=refs)
|
||||
return refs
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user