feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4)
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped
A note's relevance is now its best chunk's similarity, everywhere: - semantic_search_notes keeps the indexed raw-distance top-k and over-fetches chunk rows (x4, composing with the x3 supersession over-fetch), then collapses to first-appearance-per-note — rows arrive distance-ordered, so first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject, write-path, gate) inherits through the one function. - list_notes semantic q swaps its join for a correlated MIN-distance subquery — the join would have repeated a long note once per matching chunk and made total count chunks. - the duplicate report groups its self-join by note pair on MIN(distance): pair similarity = closest chunk pair, and the < join now also drops cross-chunk self-pairs that would flag every long note against itself. - the write gate queries once per chunk of the candidate (capped at 8), so a note duplicating an existing record in ONE SECTION is caught — the whole-document query diluted exactly the section that mattered. Integration test now seeds a two-chunk note and pins the collapse against real pgvector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -124,6 +124,14 @@ _SUPERSESSION_PENALTY = 0.05
|
||||
# whose neighbours sit ~0.01-0.02 apart.
|
||||
_SUPERSESSION_OVERFETCH = 3
|
||||
|
||||
# Chunk rows fetched per requested result (#280). The HNSW top-k runs at CHUNK
|
||||
# grain — several chunks of one strong note can occupy consecutive ranks, and
|
||||
# each collapses into a single result. Four ranks of headroom per result keeps
|
||||
# the top-k indexed while making it effectively impossible for collapsing to
|
||||
# starve the result list: that would need every requested note to be shadowed
|
||||
# by four chunks of notes ranked above it.
|
||||
_CHUNK_OVERFETCH = 4
|
||||
|
||||
|
||||
async def _apply_supersession_penalty(
|
||||
scored: list[tuple[float, "Note"]], limit: int
|
||||
@@ -524,7 +532,9 @@ async def semantic_search_notes(
|
||||
# penalty far smaller than the window's score spread, that case
|
||||
# needs the true answer to be more than _SUPERSESSION_OVERFETCH
|
||||
# ranks down, which no observed query comes close to.
|
||||
fetch = limit * _SUPERSESSION_OVERFETCH if demote_superseded else limit
|
||||
fetch = limit * _CHUNK_OVERFETCH * (
|
||||
_SUPERSESSION_OVERFETCH if demote_superseded else 1
|
||||
)
|
||||
stmt = (
|
||||
stmt.where(distance <= max_distance)
|
||||
.order_by(distance.asc())
|
||||
@@ -535,8 +545,19 @@ async def semantic_search_notes(
|
||||
logger.warning("Failed to query note embeddings", exc_info=True)
|
||||
return []
|
||||
|
||||
# Recover similarity (1 - distance) and preserve the highest-first contract.
|
||||
scored = [(1.0 - float(dist), note) for note, dist in rows]
|
||||
# Collapse chunk rows to BEST-CHUNK-PER-NOTE (#280): rows arrive ordered by
|
||||
# distance, so the first appearance of a note is its best chunk and later
|
||||
# appearances are the same note matched less well. A note's relevance IS
|
||||
# its best section's relevance — a query about one topic of a long record
|
||||
# must find that record as strongly as if the topic were the whole record.
|
||||
# Recover similarity (1 - distance); order stays highest-first.
|
||||
scored: list[tuple[float, Note]] = []
|
||||
seen: set[int] = set()
|
||||
for note, dist in rows:
|
||||
if int(note.id) in seen:
|
||||
continue
|
||||
seen.add(int(note.id))
|
||||
scored.append((1.0 - float(dist), note))
|
||||
if not demote_superseded:
|
||||
return scored[:limit]
|
||||
return await _apply_supersession_penalty(scored, limit)
|
||||
|
||||
Reference in New Issue
Block a user