bvandeusenandClaude Fable 5
0e70a3896b
note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus chunk_text (what this vector actually encodes) and chunker_version. Migration 0077 clears the table — embeddings are derived (0067 precedent) and the old whole-document rows are indistinguishable from single-chunk notes, so the startup backfill regenerates the corpus at the new shape. The backfill is now version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds exactly the stale notes, not another wipe. upsert_note_embedding takes (title, body) and chunks internally — one path for the write path, the recurrence spawn and the backfill. The recurrence spawn's own embed call is deleted outright: create_note already embeds via embed_note (#2056), so the spawn was a second copy of the rule. An emptied record now CLEARS its stale vectors instead of leaving them findable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
48 lines
2.0 KiB
Python
48 lines
2.0 KiB
Python
from datetime import datetime, timezone
|
|
|
|
from pgvector.sqlalchemy import Vector
|
|
from sqlalchemy import DateTime, ForeignKey, Integer, Text
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from scribe.models import Base
|
|
|
|
# bge-small-en-v1.5 produces 384-dim unit-normalized vectors. The column is a
|
|
# native pgvector `vector(384)` (see migration 0067) so similarity search runs
|
|
# as an indexed `ORDER BY embedding <=> :q LIMIT k` in Postgres rather than a
|
|
# full-table Python cosine scan.
|
|
EMBEDDING_DIM = 384
|
|
|
|
|
|
class NoteEmbedding(Base):
|
|
"""One embedding vector per CHUNK of a note (#280, migration 0077).
|
|
|
|
The model reads at most 512 tokens, so a single whole-document vector
|
|
permanently lost everything past ~400 words. A note now stores one row per
|
|
chunk of `embeddings.chunk_document`, and a query matches the note if it
|
|
matches ANY chunk — retrieval collapses rows to best-chunk-per-note.
|
|
A short note has exactly one row (chunk_index 0) whose text is the
|
|
historical `title\\nbody` shape.
|
|
"""
|
|
|
|
__tablename__ = "note_embeddings"
|
|
|
|
note_id: Mapped[int] = mapped_column(
|
|
Integer,
|
|
ForeignKey("notes.id", ondelete="CASCADE"),
|
|
primary_key=True,
|
|
)
|
|
chunk_index: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
user_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
|
|
embedding: Mapped[list] = mapped_column(Vector(EMBEDDING_DIM), nullable=False)
|
|
# Exactly what this vector encodes — inspectable when a ranking surprises,
|
|
# and the hook for surfacing WHICH section matched, later.
|
|
chunk_text: Mapped[str] = mapped_column(Text, nullable=False)
|
|
# embeddings.CHUNKER_VERSION at write time. The startup backfill re-embeds
|
|
# any note whose rows carry a stale version — shape changes become a
|
|
# version bump instead of a table wipe.
|
|
chunker_version: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True),
|
|
default=lambda: datetime.now(timezone.utc),
|
|
)
|