bvandeusenandClaude Fable 5
0e70a3896b
note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus chunk_text (what this vector actually encodes) and chunker_version. Migration 0077 clears the table — embeddings are derived (0067 precedent) and the old whole-document rows are indistinguishable from single-chunk notes, so the startup backfill regenerates the corpus at the new shape. The backfill is now version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds exactly the stale notes, not another wipe. upsert_note_embedding takes (title, body) and chunks internally — one path for the write path, the recurrence spawn and the backfill. The recurrence spawn's own embed call is deleted outright: create_note already embeds via embed_note (#2056), so the spawn was a second copy of the rule. An emptied record now CLEARS its stale vectors instead of leaving them findable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
55 lines
2.3 KiB
Python
55 lines
2.3 KiB
Python
"""Chunked embeddings: one note_embeddings row per chunk (#280)
|
|
|
|
Revision ID: 0077
|
|
Revises: 0076
|
|
Create Date: 2026-08-09
|
|
|
|
The embedding model reads at most 512 tokens and fastembed truncates the rest
|
|
silently, so the old one-row-per-note shape permanently lost everything past
|
|
~400 words of a record. A note now stores one row per chunk of
|
|
`embeddings.chunk_document`: PK (note_id, chunk_index), plus the chunk's text
|
|
(inspectability + future "matched section" surfacing) and the chunker version
|
|
that produced it (so later shape changes re-embed by version comparison
|
|
instead of repeating this wipe).
|
|
|
|
Embeddings are DERIVED data (0067 precedent): rows are cleared here and the
|
|
startup backfill regenerates the whole corpus at the new shape on next boot.
|
|
The HNSW index is untouched — it indexes chunk rows exactly as it indexed
|
|
note rows.
|
|
"""
|
|
from alembic import op
|
|
|
|
|
|
revision = "0077"
|
|
down_revision = "0076"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
# Derived data — the version-aware startup backfill re-embeds everything
|
|
# at the chunked shape. Old whole-document rows would be indistinguishable
|
|
# from properly-chunked single-chunk notes, so they cannot be carried over.
|
|
op.execute("DELETE FROM note_embeddings")
|
|
|
|
# Empty table, so NOT NULL columns need no defaults and the PK swap is
|
|
# instant.
|
|
op.execute("ALTER TABLE note_embeddings ADD COLUMN chunk_index integer NOT NULL")
|
|
op.execute("ALTER TABLE note_embeddings ADD COLUMN chunk_text text NOT NULL")
|
|
op.execute("ALTER TABLE note_embeddings ADD COLUMN chunker_version integer NOT NULL")
|
|
op.execute("ALTER TABLE note_embeddings DROP CONSTRAINT note_embeddings_pkey")
|
|
op.execute(
|
|
"ALTER TABLE note_embeddings ADD PRIMARY KEY (note_id, chunk_index)"
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
# Same reasoning in reverse: chunk rows make no sense to a whole-document
|
|
# reader, so clear and let the old backfill regenerate.
|
|
op.execute("DELETE FROM note_embeddings")
|
|
op.execute("ALTER TABLE note_embeddings DROP CONSTRAINT note_embeddings_pkey")
|
|
op.execute("ALTER TABLE note_embeddings DROP COLUMN chunk_index")
|
|
op.execute("ALTER TABLE note_embeddings DROP COLUMN chunk_text")
|
|
op.execute("ALTER TABLE note_embeddings DROP COLUMN chunker_version")
|
|
op.execute("ALTER TABLE note_embeddings ADD PRIMARY KEY (note_id)")
|