feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4)
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped

A note's relevance is now its best chunk's similarity, everywhere:

- semantic_search_notes keeps the indexed raw-distance top-k and over-fetches
  chunk rows (x4, composing with the x3 supersession over-fetch), then
  collapses to first-appearance-per-note — rows arrive distance-ordered, so
  first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject,
  write-path, gate) inherits through the one function.
- list_notes semantic q swaps its join for a correlated MIN-distance
  subquery — the join would have repeated a long note once per matching chunk
  and made total count chunks.
- the duplicate report groups its self-join by note pair on MIN(distance):
  pair similarity = closest chunk pair, and the < join now also drops
  cross-chunk self-pairs that would flag every long note against itself.
- the write gate queries once per chunk of the candidate (capped at 8), so a
  note duplicating an existing record in ONE SECTION is caught — the
  whole-document query diluted exactly the section that mattered.

Integration test now seeds a two-chunk note and pins the collapse against
real pgvector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-08-08 23:51:01 -04:00
co-authored by Claude Fable 5
parent 0e70a3896b
commit 041d8defbc
6 changed files with 164 additions and 51 deletions
+24 -3
View File
@@ -124,6 +124,14 @@ _SUPERSESSION_PENALTY = 0.05
# whose neighbours sit ~0.01-0.02 apart.
_SUPERSESSION_OVERFETCH = 3
# Chunk rows fetched per requested result (#280). The HNSW top-k runs at CHUNK
# grain — several chunks of one strong note can occupy consecutive ranks, and
# each collapses into a single result. Four ranks of headroom per result keeps
# the top-k indexed while making it effectively impossible for collapsing to
# starve the result list: that would need every requested note to be shadowed
# by four chunks of notes ranked above it.
_CHUNK_OVERFETCH = 4
async def _apply_supersession_penalty(
scored: list[tuple[float, "Note"]], limit: int
@@ -524,7 +532,9 @@ async def semantic_search_notes(
# penalty far smaller than the window's score spread, that case
# needs the true answer to be more than _SUPERSESSION_OVERFETCH
# ranks down, which no observed query comes close to.
fetch = limit * _SUPERSESSION_OVERFETCH if demote_superseded else limit
fetch = limit * _CHUNK_OVERFETCH * (
_SUPERSESSION_OVERFETCH if demote_superseded else 1
)
stmt = (
stmt.where(distance <= max_distance)
.order_by(distance.asc())
@@ -535,8 +545,19 @@ async def semantic_search_notes(
logger.warning("Failed to query note embeddings", exc_info=True)
return []
# Recover similarity (1 - distance) and preserve the highest-first contract.
scored = [(1.0 - float(dist), note) for note, dist in rows]
# Collapse chunk rows to BEST-CHUNK-PER-NOTE (#280): rows arrive ordered by
# distance, so the first appearance of a note is its best chunk and later
# appearances are the same note matched less well. A note's relevance IS
# its best section's relevance — a query about one topic of a long record
# must find that record as strongly as if the topic were the whole record.
# Recover similarity (1 - distance); order stays highest-first.
scored: list[tuple[float, Note]] = []
seen: set[int] = set()
for note, dist in rows:
if int(note.id) in seen:
continue
seen.add(int(note.id))
scored.append((1.0 - float(dist), note))
if not demote_superseded:
return scored[:limit]
return await _apply_supersession_penalty(scored, limit)