feat(retrieval): every semantic search hands on the passage that matched
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Failing after 1m9s
CI & Build / Build & push image (push) Skipped
#4243 fixed one door. Scribe has three semantic searches over three chunk tables, and all three collapsed chunk rows to the best one per record — each of them KNEW which passage earned the hit, and each dropped it. Every surface downstream then previewed the head of the document instead: a span the search had already scored lower, with nothing saying so. Mechanism, one place: - embeddings.record_best_chunk publishes {id: {index, text}} into `report`. Carried in `report`, NOT the return value: all three return list[tuple[float, Record]] and ~30 sites unpack that pair (lesson #4207). - semantic_search_rules and semantic_search_milestones now select chunk_index/chunk_text and publish the winner, as notes already did. semantic_search_milestones gains `report`, which it had no way to take. - services/text.matched_excerpt is the one choice of span, and excerpt_fields the one result block. Doors keep their own field names — the web renders `snippet`, MCP returns `excerpt` — because renaming a field a frontend reads is a different change from fixing what goes in it. Surfaces: - knowledge.query_knowledge, whose own comment calls it "the human's MAIN search surface", was `(note.body or "")[:200]` on every row alike. Now the matched passage on a search, the opening on a browse, and `snippet_is` saying which. KnowledgeView renders that snippet, so this was live. - search(content_type='milestone') gains `matched` — the plan body stays out, but the passage that matched comes along, because recognising a plan means recognising the part you asked about and a description written at the start need not mention it. - The auto-inject menu and the write-path prior-art menu put the passage under their line. Both were title-only, which answers "does this apply?" for a lesson or snippet (the trigger is IN the title) and not at all for an issue or dev-log. No fallback to the body's opening: on a menu that is preamble dressed as a reason, and once indented it cannot be told apart. Left alone deliberately: the rule arms. A rule hint already renders the rule's TRIGGER, which is written to answer exactly "does this apply to me" and beats a matched chunk at it; and that line's budget was measured at #3851. Adding a passage there would duplicate the trigger and spend the budget twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -40,12 +40,31 @@ from scribe.services.retrieval_surfaces import (
|
||||
)
|
||||
from scribe.services.retrieval_telemetry import record_retrieval
|
||||
from scribe.services.settings import get_setting
|
||||
from scribe.services.text import elide
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Defensive cap below Claude Code's 10k additionalContext limit.
|
||||
_MAX_CHARS = 9000
|
||||
|
||||
# Max chars of the matched passage shown under an injected menu line.
|
||||
#
|
||||
# The menu used to be titles alone, on the reasoning that its job is AWARENESS —
|
||||
# make the agent know the record exists and reach for it, not dump it. That
|
||||
# holds for a lesson or a snippet, whose title carries its trigger by
|
||||
# construction ("what — when it applies"). It does not hold for an issue, a
|
||||
# dev-log or a plain note, where the title is a headline and the reason this
|
||||
# record matched is a sentence somewhere inside it. The reader was being asked
|
||||
# "is this worth opening?" and handed the one part of the record guaranteed not
|
||||
# to answer it.
|
||||
#
|
||||
# 200 rather than more because this is a menu: eight lines at 200 is ~1.6KB,
|
||||
# which buys the decision without turning an awareness push into a dump. It is
|
||||
# the PASSAGE THAT MATCHED, not the record's opening — the search already knows
|
||||
# which one that is and used to throw it away (#4243) — so 200 characters here
|
||||
# are worth far more than 200 characters of preamble.
|
||||
_MENU_PASSAGE_CHARS = 200
|
||||
|
||||
# Max chars of a Process body to fold into the auto-surface description.
|
||||
_PROC_PREVIEW_CHARS = 200
|
||||
|
||||
@@ -1095,6 +1114,10 @@ async def build_autoinject_hint(
|
||||
# which is worse than never having surfaced it. One query for the whole menu.
|
||||
stale = await superseded_ids([int(n.id) for _s, n in kept])
|
||||
|
||||
# From THIS arm's own search (`_rep_ai`), so a chunk is only ever paired
|
||||
# with the query that actually matched it.
|
||||
menu_chunks = _rep_ai.get("best_chunk") or {}
|
||||
|
||||
note_ids: list[int] = []
|
||||
for score, note in kept:
|
||||
note_ids.append(int(note.id))
|
||||
@@ -1113,6 +1136,16 @@ async def build_autoinject_hint(
|
||||
who = owners.get(int(note.user_id)) or "another user"
|
||||
line += f" — shared by {who}, treat as a suggestion"
|
||||
lines.append(line)
|
||||
# The passage that earned the line, indented under it. Absent when the
|
||||
# record has no stored chunk — an un-embedded row, or the reserved
|
||||
# lesson and reuse slots, which are fetched by their own queries and so
|
||||
# are not in this search's report. No fallback to the body's opening:
|
||||
# on a menu that would be a line of preamble dressed as a reason, and a
|
||||
# reader cannot tell the two apart once they are indented identically.
|
||||
passage = (menu_chunks.get(int(note.id)) or {}).get("text") or ""
|
||||
if passage.strip():
|
||||
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
|
||||
lines.append(f"> ↳ {short}")
|
||||
|
||||
# Records what SURVIVED the margin gate, not what the ranker returned — the
|
||||
# menu the agent actually saw. retrieval_logs already holds the full
|
||||
@@ -2035,6 +2068,11 @@ async def build_write_path_hint(
|
||||
# floor. Applying the floor after this would throw away the best queries.
|
||||
if query:
|
||||
query = concept_query(query) or query
|
||||
# Declared out here because the search below is conditional — this arm runs
|
||||
# only when the menu has room AND a query survived the floor. The render
|
||||
# loop is not conditional, so it needs something to read either way, and an
|
||||
# empty mapping means every line falls back to its title alone.
|
||||
wp_chunks: dict[int, dict] = {}
|
||||
if remaining > 0 and query:
|
||||
t0 = time.perf_counter()
|
||||
# Pulled-and-already-listed ids stay in the query (as evidence for
|
||||
@@ -2089,6 +2127,7 @@ async def build_write_path_hint(
|
||||
scope="browse",
|
||||
report=_rep_wp,
|
||||
)
|
||||
wp_chunks = _rep_wp.get("best_chunk") or {}
|
||||
resembles = {
|
||||
int(note.id): float(score) for score, note in hits
|
||||
if int(note.id) in pulled
|
||||
@@ -2337,6 +2376,15 @@ async def build_write_path_hint(
|
||||
for item, marker, owner, foreign_lang in rendered:
|
||||
note_ids.append(int(item["id"]))
|
||||
lines.append(_prior_art_line(item, marker, owner, foreign_lang))
|
||||
# Only the semantically-matched lines carry a passage. The records-this-
|
||||
# file lines came from a LOCATION lookup — nothing was matched, so there
|
||||
# is no matching passage and the body's opening would be a fabricated
|
||||
# reason. Absence here is meaningful: a line with no passage under it is
|
||||
# one that earned its place by where it lives, not by what it says.
|
||||
passage = (wp_chunks.get(int(item["id"])) or {}).get("text") or ""
|
||||
if passage.strip():
|
||||
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
|
||||
lines.append(f"> ↳ {short}")
|
||||
|
||||
if stamped:
|
||||
lines.append(_stamp_line(path, stamped))
|
||||
|
||||
Reference in New Issue
Block a user