feat(retrieval): every semantic search hands on the passage that matched
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped

#4243 fixed one door. Scribe has three semantic searches over three chunk
tables, and all three collapsed chunk rows to the best one per record — each
of them KNEW which passage earned the hit, and each dropped it. Every surface
downstream then previewed the head of the document instead: a span the search
had already scored lower, with nothing saying so.

Mechanism, one place:
  - embeddings.record_best_chunk publishes {id: {index, text}} into `report`.
    Carried in `report`, NOT the return value: all three return
    list[tuple[float, Record]] and ~30 sites unpack that pair (lesson #4207).
  - semantic_search_rules and semantic_search_milestones now select
    chunk_index/chunk_text and publish the winner, as notes already did.
    semantic_search_milestones gains `report`, which it had no way to take.
  - services/text.matched_excerpt is the one choice of span, and
    excerpt_fields the one result block. Doors keep their own field names —
    the web renders `snippet`, MCP returns `excerpt` — because renaming a
    field a frontend reads is a different change from fixing what goes in it.

Surfaces:
  - knowledge.query_knowledge, whose own comment calls it "the human's MAIN
    search surface", was `(note.body or "")[:200]` on every row alike. Now the
    matched passage on a search, the opening on a browse, and `snippet_is`
    saying which. KnowledgeView renders that snippet, so this was live.
  - search(content_type='milestone') gains `matched` — the plan body stays
    out, but the passage that matched comes along, because recognising a plan
    means recognising the part you asked about and a description written at
    the start need not mention it.
  - The auto-inject menu and the write-path prior-art menu put the passage
    under their line. Both were title-only, which answers "does this apply?"
    for a lesson or snippet (the trigger is IN the title) and not at all for
    an issue or dev-log. No fallback to the body's opening: on a menu that is
    preamble dressed as a reason, and once indented it cannot be told apart.

Left alone deliberately: the rule arms. A rule hint already renders the rule's
TRIGGER, which is written to answer exactly "does this apply to me" and beats
a matched chunk at it; and that line's budget was measured at #3851. Adding a
passage there would duplicate the trigger and spend the budget twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-21 09:44:10 -04:00
co-authored by Claude Opus 5
parent 6abedb0168
commit 253fb974f3
8 changed files with 527 additions and 51 deletions
+29 -3
View File
@@ -25,6 +25,7 @@ from scribe.models import async_session
from scribe.models.note import Note
from scribe.models.base import iso
from scribe.services.access import browsable_notes_clause, readable_notes_clause
from scribe.services.text import excerpt_fields
logger = logging.getLogger(__name__)
@@ -211,12 +212,30 @@ def _verification_clause(value: str):
)
def _note_to_item(note: Note) -> dict:
def _note_to_item(note: Note, chunks: dict[int, dict] | None = None) -> dict:
"""One row for a browse or search listing.
`chunks` is `report["best_chunk"]` from a semantic search, when this row
came from one. With it the card shows the passage that MATCHED; without it
— a plain listing, where nothing was matched and so no span is better than
any other — it shows the record's opening. `snippet_is` says which, on
every row, so the two never have to be told apart by guessing.
This was `(note.body or "")[:_SNIPPET_LEN]` on every row alike: the head of
the document, on the human's main search surface, with nothing marking the
cut. A record matched on its sixth paragraph was shown its first, which the
search had already scored lower (#4243).
"""
item: dict = {
"id": note.id,
"note_type": note.note_type or "note",
"title": note.title,
"snippet": (note.body or "")[:_SNIPPET_LEN],
**excerpt_fields(
note.body or "",
(chunks or {}).get(int(note.id)),
_SNIPPET_LEN,
key="snippet",
),
"tags": note.tags or [],
"project_id": note.project_id,
# These lists now include records shared with the caller, so the client
@@ -507,6 +526,9 @@ async def _semantic_knowledge_search(
# record would be findable by wording and invisible by meaning — which is the
# case a semantic search exists to serve.
semantic_notes: list[Note] = []
# Filled by the search below; stays empty when the embedder is down or the
# call raises, in which case every row falls back to its opening.
_semantic_report: dict = {}
try:
from scribe.services.embeddings import (
INTERACTIVE_SEARCH_THRESHOLD,
@@ -519,6 +541,7 @@ async def _semantic_knowledge_search(
user_id=user_id,
scope="read",
query=q,
report=_semantic_report,
limit=min(200, limit * 4),
# The shared interactive floor — this was a bare `0.3` while
# routes/search.py had the same number as a commented constant, the
@@ -571,7 +594,10 @@ async def _semantic_knowledge_search(
total = len(merged)
page_items = merged[offset: offset + limit]
return [_note_to_item(n) for n in page_items], total
return [
_note_to_item(n, _semantic_report.get("best_chunk"))
for n in page_items
], total
async def get_knowledge_tags(user_id: int, note_type: str | None = None) -> list[str]: