feat(embeddings): every vector records the model whose space it lives in (#4132)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Canceled after 27s

The four embedding tables stamped chunker_version but not the model, and
vector(384) is a width, not an identity: a same-width model swap would
write a second geometry beside the first with no error.

- embedding_model on note/rule/milestone/system embeddings (0109; existing
  rows stamped with the only model any install has ever run).
- Every write stamps EMBEDDING_MODEL; every backfill's "current" test is
  is_current_stamp(), both halves of calibration_stamp().
- migrate_floor refuses while any row its surface searches is off the live
  model, before sampling: re-embed, then migrate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-23 19:02:30 -04:00
co-authored by Claude Opus 5.5
parent 11b286d786
commit f4e9cd429b
10 changed files with 178 additions and 10 deletions
@@ -58,9 +58,11 @@ import logging
from sqlalchemy import select
from scribe.models import async_session
from scribe.models.embedding import NoteEmbedding, RuleEmbedding
from scribe.models.retrieval_log import RetrievalLog
from scribe.services.embeddings import (
calibration_stamp,
rows_off_the_live_model,
semantic_search_notes,
semantic_search_rules,
)
@@ -117,6 +119,19 @@ _RESCORERS = {
"report_preference": lambda u, q, p: _rescore_rules(u, q, p, "preference"),
}
# The embedding table each surface's re-scorer reads. A migration from a corpus
# that is still part-way through a model change re-scores against a blend of
# two geometries and writes a floor computed from it, with a confident reason
# attached (#4132) — so the corpus has to be wholly in the live space first.
_CORPUS = {
"auto_inject": NoteEmbedding,
"write_path": NoteEmbedding,
"write_path_rule": RuleEmbedding,
"pre_tool_rule": RuleEmbedding,
"prompt_rule": RuleEmbedding,
"report_preference": RuleEmbedding,
}
def _floor_admitting(scores: list[float], fraction: float) -> float:
"""The floor that admits `fraction` of `scores`, on this scale.
@@ -161,6 +176,21 @@ async def migrate_floor(
"arm's own corpus filters."
)
off_model = await rows_off_the_live_model(_CORPUS[surface])
if off_model:
# Refused before sampling anything. The order of operations — re-embed,
# THEN migrate — used to be held only in the operator's memory.
stamp = calibration_stamp()
return {
"surface": surface, "migrated": False,
"why": f"{off_model} embedding row(s) this surface searches are not "
f"yet in {stamp['embedding_model']}'s space. Re-scoring now "
"would measure a blend of two models; let the startup "
"backfill finish re-embedding, then migrate",
"rows_off_model": off_model,
"calibration": stamp,
}
old_floor = await floor_for(user_id, surface)
async with async_session() as session:
rows = (await session.execute(