feat(retrieval): every semantic search hands on the passage that matched
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped

#4243 fixed one door. Scribe has three semantic searches over three chunk
tables, and all three collapsed chunk rows to the best one per record — each
of them KNEW which passage earned the hit, and each dropped it. Every surface
downstream then previewed the head of the document instead: a span the search
had already scored lower, with nothing saying so.

Mechanism, one place:
  - embeddings.record_best_chunk publishes {id: {index, text}} into `report`.
    Carried in `report`, NOT the return value: all three return
    list[tuple[float, Record]] and ~30 sites unpack that pair (lesson #4207).
  - semantic_search_rules and semantic_search_milestones now select
    chunk_index/chunk_text and publish the winner, as notes already did.
    semantic_search_milestones gains `report`, which it had no way to take.
  - services/text.matched_excerpt is the one choice of span, and
    excerpt_fields the one result block. Doors keep their own field names —
    the web renders `snippet`, MCP returns `excerpt` — because renaming a
    field a frontend reads is a different change from fixing what goes in it.

Surfaces:
  - knowledge.query_knowledge, whose own comment calls it "the human's MAIN
    search surface", was `(note.body or "")[:200]` on every row alike. Now the
    matched passage on a search, the opening on a browse, and `snippet_is`
    saying which. KnowledgeView renders that snippet, so this was live.
  - search(content_type='milestone') gains `matched` — the plan body stays
    out, but the passage that matched comes along, because recognising a plan
    means recognising the part you asked about and a description written at
    the start need not mention it.
  - The auto-inject menu and the write-path prior-art menu put the passage
    under their line. Both were title-only, which answers "does this apply?"
    for a lesson or snippet (the trigger is IN the title) and not at all for
    an issue or dev-log. No fallback to the body's opening: on a menu that is
    preamble dressed as a reason, and once indented it cannot be told apart.

Left alone deliberately: the rule arms. A rule hint already renders the rule's
TRIGGER, which is written to answer exactly "does this apply to me" and beats
a matched chunk at it; and that line's budget was measured at #3851. Adding a
passage there would duplicate the trigger and spend the budget twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-21 09:44:10 -04:00
co-authored by Claude Opus 5
parent 6abedb0168
commit 253fb974f3
8 changed files with 527 additions and 51 deletions
+48
View File
@@ -40,12 +40,31 @@ from scribe.services.retrieval_surfaces import (
)
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.settings import get_setting
from scribe.services.text import elide
logger = logging.getLogger(__name__)
# Defensive cap below Claude Code's 10k additionalContext limit.
_MAX_CHARS = 9000
# Max chars of the matched passage shown under an injected menu line.
#
# The menu used to be titles alone, on the reasoning that its job is AWARENESS —
# make the agent know the record exists and reach for it, not dump it. That
# holds for a lesson or a snippet, whose title carries its trigger by
# construction ("what — when it applies"). It does not hold for an issue, a
# dev-log or a plain note, where the title is a headline and the reason this
# record matched is a sentence somewhere inside it. The reader was being asked
# "is this worth opening?" and handed the one part of the record guaranteed not
# to answer it.
#
# 200 rather than more because this is a menu: eight lines at 200 is ~1.6KB,
# which buys the decision without turning an awareness push into a dump. It is
# the PASSAGE THAT MATCHED, not the record's opening — the search already knows
# which one that is and used to throw it away (#4243) — so 200 characters here
# are worth far more than 200 characters of preamble.
_MENU_PASSAGE_CHARS = 200
# Max chars of a Process body to fold into the auto-surface description.
_PROC_PREVIEW_CHARS = 200
@@ -1095,6 +1114,10 @@ async def build_autoinject_hint(
# which is worse than never having surfaced it. One query for the whole menu.
stale = await superseded_ids([int(n.id) for _s, n in kept])
# From THIS arm's own search (`_rep_ai`), so a chunk is only ever paired
# with the query that actually matched it.
menu_chunks = _rep_ai.get("best_chunk") or {}
note_ids: list[int] = []
for score, note in kept:
note_ids.append(int(note.id))
@@ -1113,6 +1136,16 @@ async def build_autoinject_hint(
who = owners.get(int(note.user_id)) or "another user"
line += f" — shared by {who}, treat as a suggestion"
lines.append(line)
# The passage that earned the line, indented under it. Absent when the
# record has no stored chunk — an un-embedded row, or the reserved
# lesson and reuse slots, which are fetched by their own queries and so
# are not in this search's report. No fallback to the body's opening:
# on a menu that would be a line of preamble dressed as a reason, and a
# reader cannot tell the two apart once they are indented identically.
passage = (menu_chunks.get(int(note.id)) or {}).get("text") or ""
if passage.strip():
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
lines.append(f"> ↳ {short}")
# Records what SURVIVED the margin gate, not what the ranker returned — the
# menu the agent actually saw. retrieval_logs already holds the full
@@ -2035,6 +2068,11 @@ async def build_write_path_hint(
# floor. Applying the floor after this would throw away the best queries.
if query:
query = concept_query(query) or query
# Declared out here because the search below is conditional — this arm runs
# only when the menu has room AND a query survived the floor. The render
# loop is not conditional, so it needs something to read either way, and an
# empty mapping means every line falls back to its title alone.
wp_chunks: dict[int, dict] = {}
if remaining > 0 and query:
t0 = time.perf_counter()
# Pulled-and-already-listed ids stay in the query (as evidence for
@@ -2089,6 +2127,7 @@ async def build_write_path_hint(
scope="browse",
report=_rep_wp,
)
wp_chunks = _rep_wp.get("best_chunk") or {}
resembles = {
int(note.id): float(score) for score, note in hits
if int(note.id) in pulled
@@ -2337,6 +2376,15 @@ async def build_write_path_hint(
for item, marker, owner, foreign_lang in rendered:
note_ids.append(int(item["id"]))
lines.append(_prior_art_line(item, marker, owner, foreign_lang))
# Only the semantically-matched lines carry a passage. The records-this-
# file lines came from a LOCATION lookup — nothing was matched, so there
# is no matching passage and the body's opening would be a fabricated
# reason. Absence here is meaningful: a line with no passage under it is
# one that earned its place by where it lives, not by what it says.
passage = (wp_chunks.get(int(item["id"])) or {}).get("text") or ""
if passage.strip():
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
lines.append(f"> ↳ {short}")
if stamped:
lines.append(_stamp_line(path, stamped))