fix(retrieval): every semantic search scopes first, then ranks - the note, milestone and system searches join the rule search on one shared shape, _rank_scoped (#4961)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m0s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 27s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m0s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 27s
Ordered straight off a *_embeddings table, the planner walks the HNSW index, takes ~ef_search (40) nearest chunks across every owner and project, and only then applies the scope: an in-scope record behind 40 nearer ones the caller cannot see was silently dropped. #4958 fixed the rule search alone; the note, milestone and system searches kept the fault. _scoped_chunks builds the in-scope chunks with their distance; _rank_scoped materializes them as a CTE and ranks exactly. All four searches go through it, and the row shape is unchanged, so callers and mocks are untouched. Tests: a structural guard that every semantic_search_* ranks through _rank_scoped and orders nothing itself (with a replay of the old shape), a compiled-SQL check of the MATERIALIZED CTE, and integration crowd tests - 80 nearer out-of-scope records - for notes (another user; the reader own other project), milestones and systems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,9 +7,10 @@ from sqlalchemy.orm import Mapped, mapped_column
|
||||
from scribe.models import Base
|
||||
|
||||
# bge-small-en-v1.5 produces 384-dim unit-normalized vectors. The column is a
|
||||
# native pgvector `vector(384)` (see migration 0067) so similarity search runs
|
||||
# as an indexed `ORDER BY embedding <=> :q LIMIT k` in Postgres rather than a
|
||||
# full-table Python cosine scan.
|
||||
# native pgvector `vector(384)` (see migration 0067) so similarity is ranked in
|
||||
# Postgres by `<=>` rather than by a full-table Python cosine scan. The semantic
|
||||
# searches rank the CALLER'S in-scope chunks exactly rather than through the
|
||||
# HNSW index, which ranks before it filters (#4961; embeddings._rank_scoped).
|
||||
EMBEDDING_DIM = 384
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user