The embedding model reads at most 512 tokens and fastembed truncates the rest silently, so the one-vector-per-note shape permanently lost everything past ~400 words of a record — for a long dev-log, three quarters of its content could never influence search. Operator's call: no authoring bandaid, build the infrastructure right.
The shape — chunk_document(title, body): fence-aware markdown-heading sections, small sections merged, oversize sections split at paragraph boundaries, title anchoring every chunk, section headings repeated on continuation pieces. A record that fits the window yields exactly one chunk identical to the historical shape — snippets and reference notes are byte-for-byte unaffected.
Storage — migration 0077: note_embeddings PK becomes (note_id, chunk_index) + chunk_text + chunker_version; table cleared (derived data, 0067 precedent). The startup backfill is version-aware: future shape changes are a CHUNKER_VERSION bump that re-embeds exactly the stale notes, not another wipe.
Retrieval — best chunk wins, everywhere: semantic_search_notes over-fetches chunk rows on the HNSW index (×4, composing with the ×3 supersession over-fetch) and collapses to best-chunk-per-note, inherited by every ranked consumer; list-q swaps its join for a correlated MIN-distance subquery; the duplicate report groups its self-join by note pair on MIN(distance); the write gate queries per candidate chunk (capped at 8), so a note duplicating an existing record in one section is now caught.
Deploy notes: migration 0077 runs on boot; first boot then re-embeds the whole corpus in the background (in-process model, minutes). Semantic search degrades gracefully to keyword during the backfill window.
CI green on head 23cb396 (run 3553), including the integration lane running 0077 + a two-chunk collapse test against real pgvector.
The embedding model reads at most 512 tokens and fastembed truncates the rest silently, so the one-vector-per-note shape permanently lost everything past ~400 words of a record — for a long dev-log, three quarters of its content could never influence search. Operator's call: no authoring bandaid, build the infrastructure right.
**The shape** — `chunk_document(title, body)`: fence-aware markdown-heading sections, small sections merged, oversize sections split at paragraph boundaries, title anchoring every chunk, section headings repeated on continuation pieces. A record that fits the window yields exactly one chunk identical to the historical shape — snippets and reference notes are byte-for-byte unaffected.
**Storage** — migration **0077**: `note_embeddings` PK becomes `(note_id, chunk_index)` + `chunk_text` + `chunker_version`; table cleared (derived data, 0067 precedent). The startup backfill is version-aware: future shape changes are a `CHUNKER_VERSION` bump that re-embeds exactly the stale notes, not another wipe.
**Retrieval** — best chunk wins, everywhere: `semantic_search_notes` over-fetches chunk rows on the HNSW index (×4, composing with the ×3 supersession over-fetch) and collapses to best-chunk-per-note, inherited by every ranked consumer; list-q swaps its join for a correlated MIN-distance subquery; the duplicate report groups its self-join by note pair on MIN(distance); the write gate queries per candidate chunk (capped at 8), so a note duplicating an existing record in one section is now caught.
**Deploy notes**: migration 0077 runs on boot; first boot then re-embeds the whole corpus in the background (in-process model, minutes). Semantic search degrades gracefully to keyword during the backfill window.
CI green on head `23cb396` (run 3553), including the integration lane running 0077 + a two-chunk collapse test against real pgvector.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
bge-small reads 512 tokens and fastembed truncates silently, so a single
whole-document vector permanently lost everything past ~400 words. The new
shape: split at markdown headings (fence-aware), merge small sections, split
oversize ones at paragraph boundaries, title-anchor every chunk, repeat the
section heading on continuation pieces. A record that fits the window yields
exactly one chunk identical to the historical title\nbody shape, so the
corpus's sharpest records are byte-for-byte unaffected. CHUNKER_VERSION added
so later shape changes re-embed by version comparison instead of a table wipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus
chunk_text (what this vector actually encodes) and chunker_version. Migration
0077 clears the table — embeddings are derived (0067 precedent) and the old
whole-document rows are indistinguishable from single-chunk notes, so the
startup backfill regenerates the corpus at the new shape. The backfill is now
version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds
exactly the stale notes, not another wipe.
upsert_note_embedding takes (title, body) and chunks internally — one path
for the write path, the recurrence spawn and the backfill. The recurrence
spawn's own embed call is deleted outright: create_note already embeds via
embed_note (#2056), so the spawn was a second copy of the rule. An emptied
record now CLEARS its stale vectors instead of leaving them findable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
A note's relevance is now its best chunk's similarity, everywhere:
- semantic_search_notes keeps the indexed raw-distance top-k and over-fetches
chunk rows (x4, composing with the x3 supersession over-fetch), then
collapses to first-appearance-per-note — rows arrive distance-ordered, so
first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject,
write-path, gate) inherits through the one function.
- list_notes semantic q swaps its join for a correlated MIN-distance
subquery — the join would have repeated a long note once per matching chunk
and made total count chunks.
- the duplicate report groups its self-join by note pair on MIN(distance):
pair similarity = closest chunk pair, and the < join now also drops
cross-chunk self-pairs that would flag every long note against itself.
- the write gate queries once per chunk of the candidate (capped at 8), so a
note duplicating an existing record in ONE SECTION is caught — the
whole-document query diluted exactly the section that mattered.
Integration test now seeds a two-chunk note and pins the collapse against
real pgvector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The embedding model reads at most 512 tokens and fastembed truncates the rest silently, so the one-vector-per-note shape permanently lost everything past ~400 words of a record — for a long dev-log, three quarters of its content could never influence search. Operator's call: no authoring bandaid, build the infrastructure right.
The shape —
chunk_document(title, body): fence-aware markdown-heading sections, small sections merged, oversize sections split at paragraph boundaries, title anchoring every chunk, section headings repeated on continuation pieces. A record that fits the window yields exactly one chunk identical to the historical shape — snippets and reference notes are byte-for-byte unaffected.Storage — migration 0077:
note_embeddingsPK becomes(note_id, chunk_index)+chunk_text+chunker_version; table cleared (derived data, 0067 precedent). The startup backfill is version-aware: future shape changes are aCHUNKER_VERSIONbump that re-embeds exactly the stale notes, not another wipe.Retrieval — best chunk wins, everywhere:
semantic_search_notesover-fetches chunk rows on the HNSW index (×4, composing with the ×3 supersession over-fetch) and collapses to best-chunk-per-note, inherited by every ranked consumer; list-q swaps its join for a correlated MIN-distance subquery; the duplicate report groups its self-join by note pair on MIN(distance); the write gate queries per candidate chunk (capped at 8), so a note duplicating an existing record in one section is now caught.Deploy notes: migration 0077 runs on boot; first boot then re-embeds the whole corpus in the background (in-process model, minutes). Semantic search degrades gracefully to keyword during the backfill window.
CI green on head
23cb396(run 3553), including the integration lane running 0077 + a two-chunk collapse test against real pgvector.🤖 Generated with Claude Code
https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs