feat(retrieval): every semantic search hands on the passage that matched
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped
#4243 fixed one door. Scribe has three semantic searches over three chunk tables, and all three collapsed chunk rows to the best one per record — each of them KNEW which passage earned the hit, and each dropped it. Every surface downstream then previewed the head of the document instead: a span the search had already scored lower, with nothing saying so. Mechanism, one place: - embeddings.record_best_chunk publishes {id: {index, text}} into `report`. Carried in `report`, NOT the return value: all three return list[tuple[float, Record]] and ~30 sites unpack that pair (lesson #4207). - semantic_search_rules and semantic_search_milestones now select chunk_index/chunk_text and publish the winner, as notes already did. semantic_search_milestones gains `report`, which it had no way to take. - services/text.matched_excerpt is the one choice of span, and excerpt_fields the one result block. Doors keep their own field names — the web renders `snippet`, MCP returns `excerpt` — because renaming a field a frontend reads is a different change from fixing what goes in it. Surfaces: - knowledge.query_knowledge, whose own comment calls it "the human's MAIN search surface", was `(note.body or "")[:200]` on every row alike. Now the matched passage on a search, the opening on a browse, and `snippet_is` saying which. KnowledgeView renders that snippet, so this was live. - search(content_type='milestone') gains `matched` — the plan body stays out, but the passage that matched comes along, because recognising a plan means recognising the part you asked about and a description written at the start need not mention it. - The auto-inject menu and the write-path prior-art menu put the passage under their line. Both were title-only, which answers "does this apply?" for a lesson or snippet (the trigger is IN the title) and not at all for an issue or dev-log. No fallback to the body's opening: on a menu that is preamble dressed as a reason, and once indented it cannot be told apart. Left alone deliberately: the rule arms. A rule hint already renders the rule's TRIGGER, which is written to answer exactly "does this apply to me" and beats a matched chunk at it; and that line's budget was measured at #3851. Adding a passage there would duplicate the trigger and spend the budget twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -25,6 +25,7 @@ from scribe.models import async_session
|
||||
from scribe.models.note import Note
|
||||
from scribe.models.base import iso
|
||||
from scribe.services.access import browsable_notes_clause, readable_notes_clause
|
||||
from scribe.services.text import excerpt_fields
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -211,12 +212,30 @@ def _verification_clause(value: str):
|
||||
)
|
||||
|
||||
|
||||
def _note_to_item(note: Note) -> dict:
|
||||
def _note_to_item(note: Note, chunks: dict[int, dict] | None = None) -> dict:
|
||||
"""One row for a browse or search listing.
|
||||
|
||||
`chunks` is `report["best_chunk"]` from a semantic search, when this row
|
||||
came from one. With it the card shows the passage that MATCHED; without it
|
||||
— a plain listing, where nothing was matched and so no span is better than
|
||||
any other — it shows the record's opening. `snippet_is` says which, on
|
||||
every row, so the two never have to be told apart by guessing.
|
||||
|
||||
This was `(note.body or "")[:_SNIPPET_LEN]` on every row alike: the head of
|
||||
the document, on the human's main search surface, with nothing marking the
|
||||
cut. A record matched on its sixth paragraph was shown its first, which the
|
||||
search had already scored lower (#4243).
|
||||
"""
|
||||
item: dict = {
|
||||
"id": note.id,
|
||||
"note_type": note.note_type or "note",
|
||||
"title": note.title,
|
||||
"snippet": (note.body or "")[:_SNIPPET_LEN],
|
||||
**excerpt_fields(
|
||||
note.body or "",
|
||||
(chunks or {}).get(int(note.id)),
|
||||
_SNIPPET_LEN,
|
||||
key="snippet",
|
||||
),
|
||||
"tags": note.tags or [],
|
||||
"project_id": note.project_id,
|
||||
# These lists now include records shared with the caller, so the client
|
||||
@@ -507,6 +526,9 @@ async def _semantic_knowledge_search(
|
||||
# record would be findable by wording and invisible by meaning — which is the
|
||||
# case a semantic search exists to serve.
|
||||
semantic_notes: list[Note] = []
|
||||
# Filled by the search below; stays empty when the embedder is down or the
|
||||
# call raises, in which case every row falls back to its opening.
|
||||
_semantic_report: dict = {}
|
||||
try:
|
||||
from scribe.services.embeddings import (
|
||||
INTERACTIVE_SEARCH_THRESHOLD,
|
||||
@@ -519,6 +541,7 @@ async def _semantic_knowledge_search(
|
||||
user_id=user_id,
|
||||
scope="read",
|
||||
query=q,
|
||||
report=_semantic_report,
|
||||
limit=min(200, limit * 4),
|
||||
# The shared interactive floor — this was a bare `0.3` while
|
||||
# routes/search.py had the same number as a commented constant, the
|
||||
@@ -571,7 +594,10 @@ async def _semantic_knowledge_search(
|
||||
|
||||
total = len(merged)
|
||||
page_items = merged[offset: offset + limit]
|
||||
return [_note_to_item(n) for n in page_items], total
|
||||
return [
|
||||
_note_to_item(n, _semantic_report.get("best_chunk"))
|
||||
for n in page_items
|
||||
], total
|
||||
|
||||
|
||||
async def get_knowledge_tags(user_id: int, note_type: str | None = None) -> list[str]:
|
||||
|
||||
Reference in New Issue
Block a user