feat(embeddings): every vector records the model whose space it lives in (#4132)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Canceled after 27s

The four embedding tables stamped chunker_version but not the model, and
vector(384) is a width, not an identity: a same-width model swap would
write a second geometry beside the first with no error.

- embedding_model on note/rule/milestone/system embeddings (0109; existing
  rows stamped with the only model any install has ever run).
- Every write stamps EMBEDDING_MODEL; every backfill's "current" test is
  is_current_stamp(), both halves of calibration_stamp().
- migrate_floor refuses while any row its surface searches is off the live
  model, before sampling: re-embed, then migrate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-23 19:02:30 -04:00
co-authored by Claude Opus 5.5
parent 11b286d786
commit f4e9cd429b
10 changed files with 178 additions and 10 deletions
@@ -0,0 +1,42 @@
"""embeddings_record_their_model — a vector says whose space it lives in (#4132)
Revision ID: 0109
Revises: 0108
Create Date: 2026-09-23
Every embedding table stamps `chunker_version`, so a change to the document
shape is caught per row and re-embedded. None stamped the MODEL. The column is
`vector(384)` — a width, not an identity — so swapping bge-small for any other
384-dim model would write a second geometry beside the first with no error,
and search would go on ranking by cosines between the two, which mean nothing.
`embedding_model` is the other half of `calibration_stamp()`, stored per row on
all four tables. The startup backfill now re-embeds on either half moving.
THE BACKFILL LITERAL IS SAFE BECAUSE NO INSTALL HAS EVER CHANGED MODELS. The
name is hardcoded in `services/embeddings.py`, and every vector ever written was
written by it — so stamping every existing row with that name states a fact,
not a guess. It is frozen here rather than imported: a migration records what
was true when it ran, and a later model change must not rewrite history.
"""
from alembic import op
revision = "0109"
down_revision = "0108"
branch_labels = None
depends_on = None
_TABLES = ("note_embeddings", "rule_embeddings", "milestone_embeddings", "system_embeddings")
_MODEL = "BAAI/bge-small-en-v1.5"
def upgrade() -> None:
for table in _TABLES:
op.execute(f"ALTER TABLE {table} ADD COLUMN embedding_model text")
op.execute(f"UPDATE {table} SET embedding_model = '{_MODEL}'")
op.execute(f"ALTER TABLE {table} ALTER COLUMN embedding_model SET NOT NULL")
def downgrade() -> None:
for table in _TABLES:
op.execute(f"ALTER TABLE {table} DROP COLUMN embedding_model")