feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4)
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped
A note's relevance is now its best chunk's similarity, everywhere: - semantic_search_notes keeps the indexed raw-distance top-k and over-fetches chunk rows (x4, composing with the x3 supersession over-fetch), then collapses to first-appearance-per-note — rows arrive distance-ordered, so first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject, write-path, gate) inherits through the one function. - list_notes semantic q swaps its join for a correlated MIN-distance subquery — the join would have repeated a long note once per matching chunk and made total count chunks. - the duplicate report groups its self-join by note pair on MIN(distance): pair similarity = closest chunk pair, and the < join now also drops cross-chunk self-pairs that would flag every long note against itself. - the write gate queries once per chunk of the candidate (capped at 8), so a note duplicating an existing record in ONE SECTION is caught — the whole-document query diluted exactly the section that mattered. Integration test now seeds a two-chunk note and pins the collapse against real pgvector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -136,6 +136,36 @@ def test_a_monster_single_paragraph_is_hard_split_not_dropped():
|
||||
assert total_words == 2000
|
||||
|
||||
|
||||
# --- the read path: best chunk wins (#280 step 4) ----------------------------
|
||||
|
||||
|
||||
async def test_search_collapses_chunk_rows_to_best_chunk_per_note():
|
||||
"""Rows arrive at CHUNK grain ordered by distance; a note appearing via
|
||||
several chunks must come back ONCE, scored by its best chunk — otherwise a
|
||||
long record fills the top-k with copies of itself."""
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from scribe.services import embeddings as emb
|
||||
|
||||
note_a, note_b = MagicMock(id=1), MagicMock(id=2)
|
||||
rows = [(note_a, 0.10), (note_b, 0.20), (note_a, 0.25), (note_a, 0.30)]
|
||||
result = MagicMock()
|
||||
result.all.return_value = rows
|
||||
session, ctx = _session_ctx()
|
||||
session.execute = AsyncMock(return_value=result)
|
||||
|
||||
with (
|
||||
patch.object(emb, "async_session", return_value=ctx),
|
||||
patch.object(emb, "get_embedding", AsyncMock(return_value=[0.0] * 384)),
|
||||
):
|
||||
out = await emb.semantic_search_notes(
|
||||
1, "a query", limit=8, demote_superseded=False
|
||||
)
|
||||
|
||||
assert [note.id for _s, note in out] == [1, 2]
|
||||
assert out[0][0] == 1.0 - 0.10 # the BEST chunk's score, not a later one
|
||||
|
||||
|
||||
# --- the write path: one row per chunk (#280 step 3) -------------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user