bge-small reads 512 tokens and fastembed truncates silently, so a single
whole-document vector permanently lost everything past ~400 words. The new
shape: split at markdown headings (fence-aware), merge small sections, split
oversize ones at paragraph boundaries, title-anchor every chunk, repeat the
section heading on continuation pieces. A record that fits the window yields
exactly one chunk identical to the historical title\nbody shape, so the
corpus's sharpest records are byte-for-byte unaffected. CHUNKER_VERSION added
so later shape changes re-embed by version comparison instead of a table wipe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
