feat(retrieval): every semantic search hands on the passage that matched

#4243 fixed one door. Scribe has three semantic searches over three chunk
tables, and all three collapsed chunk rows to the best one per record — each
of them KNEW which passage earned the hit, and each dropped it. Every surface
downstream then previewed the head of the document instead: a span the search
had already scored lower, with nothing saying so.

Mechanism, one place:
  - embeddings.record_best_chunk publishes {id: {index, text}} into `report`.
    Carried in `report`, NOT the return value: all three return
    list[tuple[float, Record]] and ~30 sites unpack that pair (lesson #4207).
  - semantic_search_rules and semantic_search_milestones now select
    chunk_index/chunk_text and publish the winner, as notes already did.
    semantic_search_milestones gains `report`, which it had no way to take.
  - services/text.matched_excerpt is the one choice of span, and
    excerpt_fields the one result block. Doors keep their own field names —
    the web renders `snippet`, MCP returns `excerpt` — because renaming a
    field a frontend reads is a different change from fixing what goes in it.

Surfaces:
  - knowledge.query_knowledge, whose own comment calls it "the human's MAIN
    search surface", was `(note.body or "")[:200]` on every row alike. Now the
    matched passage on a search, the opening on a browse, and `snippet_is`
    saying which. KnowledgeView renders that snippet, so this was live.
  - search(content_type='milestone') gains `matched` — the plan body stays
    out, but the passage that matched comes along, because recognising a plan
    means recognising the part you asked about and a description written at
    the start need not mention it.
  - The auto-inject menu and the write-path prior-art menu put the passage
    under their line. Both were title-only, which answers "does this apply?"
    for a lesson or snippet (the trigger is IN the title) and not at all for
    an issue or dev-log. No fallback to the body's opening: on a menu that is
    preamble dressed as a reason, and once indented it cannot be told apart.

Left alone deliberately: the rule arms. A rule hint already renders the rule's
TRIGGER, which is written to answer exactly "does this apply to me" and beats
a matched chunk at it; and that line's budget was measured at #3851. Adding a
passage there would duplicate the trigger and spend the budget twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/embeddings.py

@@ -577,6 +577,30 @@ GLOBAL_NOTE_TYPES: tuple[str, ...] = ("lesson",)
 # write telemetry pass a dict and read `best_available_score` back out of it.
 
 
+def record_best_chunk(report: dict | None, chunks: dict[int, dict]) -> None:
+    """Publish the winning chunk per record into `report["best_chunk"]`.
+
+    Every semantic search here collapses several chunk rows to the best one per
+    record, which means each of them KNOWS which passage earned the hit — and
+    each of them used to drop it, leaving every caller to preview the head of
+    the document instead. The head is a different span, one the search has
+    already scored lower, and nothing in the result said so (#4243).
+
+    It rides in `report` rather than in the return value because all three
+    searches return `list[tuple[float, Record]]` and roughly thirty sites
+    unpack that pair; widening it would be an interface change to every one of
+    them with nothing to catch a miss (lesson #4207). `report` is already the
+    side-channel these functions use for `searched` and `best_available_score`,
+    so this adds a key to a channel callers already open.
+
+    Shape: {record_id: {"index": int, "text": str}}. A caller that passed no
+    report simply doesn't get it, and every consumer falls back to the body.
+    """
+    if report is None:
+        return
+    report["best_chunk"] = chunks
+
+
 async def semantic_search_notes(
     user_id: int,
     query: str,
@@ -823,12 +847,11 @@ async def semantic_search_notes(
         final = await _apply_supersession_penalty(scored, limit)
     # Only for what actually came back, so a caller can key straight off the
     # results without carrying chunks for records it never saw.
-    if report is not None:
-        report["best_chunk"] = {
-            int(n.id): best_chunk[int(n.id)]
-            for _s, n in final
-            if int(n.id) in best_chunk
-        }
+    record_best_chunk(report, {
+        int(n.id): best_chunk[int(n.id)]
+        for _s, n in final
+        if int(n.id) in best_chunk
+    })
     return final
 
 
@@ -1075,7 +1098,12 @@ async def semantic_search_rules(
 
         async with async_session() as session:
             rows = (await session.execute(
-                select(Rule, distance.label("distance"))
+                select(
+                    Rule,
+                    distance.label("distance"),
+                    RuleEmbedding.chunk_index,
+                    RuleEmbedding.chunk_text,
+                )
                 .select_from(RuleEmbedding)
                 .join(Rule, RuleEmbedding.rule_id == Rule.id)
                 .outerjoin(RulebookTopic, Rule.topic_id == RulebookTopic.id)
@@ -1098,11 +1126,22 @@ async def semantic_search_rules(
         return []
 
     best: dict[int, tuple[float, object]] = {}
-    for rule, dist in rows:
+    # Which chunk won, kept beside the score it won with — a rule's `why` and
+    # `how_to_apply` are long, and a caller shown only the head cannot see the
+    # clause that actually matched (#4243).
+    won: dict[int, dict] = {}
+    for rule, dist, chunk_index, chunk_text in rows:
         score = 1.0 - float(dist)
         if rule.id not in best or score > best[rule.id][0]:
             best[rule.id] = (score, rule)
+            won[int(rule.id)] = {
+                "index": int(chunk_index), "text": chunk_text or "",
+            }
     ranked = sorted(best.values(), key=lambda pair: pair[0], reverse=True)
+    kept = [pair for pair in ranked if pair[0] >= threshold][:limit]
+    record_best_chunk(report, {
+        int(r.id): won[int(r.id)] for _s, r in kept if int(r.id) in won
+    })
     if report is not None:
         # See the sibling search: absent means the search never ran (#3765).
         report["searched"] = True
@@ -1110,7 +1149,7 @@ async def semantic_search_rules(
         best = ranked[0] if ranked else None
         report["best_available_score"] = best[0] if best else None
         report["best_available_id"] = int(best[1].id) if best else None
-    return [pair for pair in ranked if pair[0] >= threshold][:limit]
+    return kept
 
 
 async def backfill_rule_embeddings() -> None:
@@ -1218,6 +1257,7 @@ async def semantic_search_milestones(
     status: str | None = None,
     limit: int = 5,
     threshold: float = _SIMILARITY_THRESHOLD,
+    report: dict | None = None,
 ) -> list[tuple[float, "Milestone"]]:
     """Return up to *limit* (score, milestone) pairs most like *query*.
 
@@ -1254,7 +1294,12 @@ async def semantic_search_milestones(
             scope = Milestone.user_id == user_id
         async with async_session() as session:
             rows = (await session.execute(
-                select(Milestone, distance.label("distance"))
+                select(
+                    Milestone,
+                    distance.label("distance"),
+                    MilestoneEmbedding.chunk_index,
+                    MilestoneEmbedding.chunk_text,
+                )
                 .select_from(MilestoneEmbedding)
                 .join(Milestone, MilestoneEmbedding.milestone_id == Milestone.id)
                 .where(
@@ -1270,12 +1315,23 @@ async def semantic_search_milestones(
         return []
 
     best: dict[int, tuple[float, object]] = {}
-    for milestone, dist in rows:
+    # A milestone's `body` IS the plan, and search results show its short
+    # `description` — so a match on the design was previewed by a sentence that
+    # need not mention it. The winning chunk is what the caller should see.
+    won: dict[int, dict] = {}
+    for milestone, dist, chunk_index, chunk_text in rows:
         score = 1.0 - float(dist)
         if milestone.id not in best or score > best[milestone.id][0]:
             best[milestone.id] = (score, milestone)
+            won[int(milestone.id)] = {
+                "index": int(chunk_index), "text": chunk_text or "",
+            }
     ranked = sorted(best.values(), key=lambda pair: pair[0], reverse=True)
-    return [pair for pair in ranked if pair[0] >= threshold][:limit]
+    kept = [pair for pair in ranked if pair[0] >= threshold][:limit]
+    record_best_chunk(report, {
+        int(m.id): won[int(m.id)] for _s, m in kept if int(m.id) in won
+    })
+    return kept
 
 
 async def backfill_milestone_embeddings() -> None: