feat(retrieval): a menu line is a name, its kind and System, and the whole passage that matched (#4364)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 58s
CI & Build / Python tests (push) Successful in 1m36s
CI & Build / Build & push image (push) Successful in 26s

Injected lines rendered a snippet's or lesson's title, which carries its
whole trigger by construction (the embedding shape) and ran past 1,500
characters -- again on every `seen` repeat. The passage under a line was
cut to 200 chars from the middle, keeping its head (the title again) and
losing where the match was.

Now, on both the prompt menu and the write-path prior-art menu:
- the line shows the record's NAME (snippet data.name / lesson subject),
  with its kind and System (`[issue (done) · Plugin & hooks]`);
- the passage is the whole matched chunk, title prefix stripped, on one
  line so the blockquote holds; a title-only match hands over the trigger;
- a `seen` record is a one-line pointer to what is already in context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-23 16:29:48 -04:00
co-authored by Claude Opus 5.5
parent 20227ebb5d
commit bb632c4196
4 changed files with 262 additions and 46 deletions
+122 -45
View File
@@ -40,6 +40,7 @@ from scribe.services.retrieval_surfaces import (
)
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.settings import get_setting
from scribe.services.systems import system_names_for
from scribe.services.text import elide
logger = logging.getLogger(__name__)
@@ -47,23 +48,71 @@ logger = logging.getLogger(__name__)
# Defensive cap below Claude Code's 10k additionalContext limit.
_MAX_CHARS = 9000
# Max chars of the matched passage shown under an injected menu line.
# WHAT A MENU LINE CARRIES (#4364): the record's NAME, its kind and System,
# and the WHOLE passage that matched. Metadata plus the evidence, rather than a
# title asked to be both.
#
# The menu used to be titles alone, on the reasoning that its job is AWARENESS —
# make the agent know the record exists and reach for it, not dump it. That
# holds for a lesson or a snippet, whose title carries its trigger by
# construction ("what — when it applies"). It does not hold for an issue, a
# dev-log or a plain note, where the title is a headline and the reason this
# record matched is a sentence somewhere inside it. The reader was being asked
# "is this worth opening?" and handed the one part of the record guaranteed not
# to answer it.
# The name, not the title. A snippet's or lesson's title is `name — when it
# applies` by construction (`embeddings.trigger_title`), because that join is
# what makes it rank on its situation. That is an EMBEDDING shape, and rendered
# as a menu line it ran to 1,500+ characters — the trigger paragraph spent
# again on every line, and again on every repeat. The trigger still arrives
# when it is what matched: it is in the chunk, and `_menu_passage` hands it
# over when the title was the whole match.
#
# 200 rather than more because this is a menu: eight lines at 200 is ~1.6KB,
# which buys the decision without turning an awareness push into a dump. It is
# the PASSAGE THAT MATCHED, not the record's opening — the search already knows
# which one that is and used to throw it away (#4243) — so 200 characters here
# are worth far more than 200 characters of preamble.
_MENU_PASSAGE_CHARS = 200
# The whole passage, not 200 characters of it. The search already chose the
# chunk that matched; the old cut kept its head and tail, and the head is the
# title every chunk is prefixed with — so the reader got the title twice and
# lost the middle, which is where the match was (lesson #4248). A chunk is at
# most ~1.4 KB (`embeddings._CHUNK_CHAR_BUDGET`), and it is shown once: a
# repeat is a one-line pointer (`_menu_seen_line`), not a second copy.
def _menu_name(title: str | None, note_type: str | None, data=None, body: str | None = "") -> str:
"""The record's name — its title without the trigger composed into it."""
title = (title or "(untitled)").replace("\n", " ").strip()
data = data if isinstance(data, dict) else {}
if note_type == "snippet":
from scribe.services.embeddings import TRIGGER_SEP
return (data.get("name") or title.partition(TRIGGER_SEP)[0]).strip() or title
if note_type == LESSON_NOTE_TYPE:
from types import SimpleNamespace
from scribe.services.embeddings import untrigger_title
from scribe.services.lessons import lesson_trigger
trigger = lesson_trigger(SimpleNamespace(data=data, body=body or ""))
return untrigger_title(title, trigger).strip() or title
return title
def _menu_passage(title: str | None, chunk_text: str | None, name: str = "") -> str:
"""The matched chunk on one line, without the title it was embedded under.
Every chunk is `title\nsection` (`embeddings.embedding_text`), so the title
prefix is stripped exactly. A chunk that WAS only the title — a short
record, or the head chunk of one — matched on the title, and for a
trigger-keyed kind the part of it the name line no longer shows is the
trigger: that is returned, because it is precisely what matched.
One line, so the menu's blockquote survives it.
"""
title = (title or "").strip()
text = (chunk_text or "").strip()
if title and text.startswith(title):
text = text[len(title):]
text = " ".join(text.split())
if not text and name and title.startswith(name) and title != name:
text = " ".join(title[len(name):].lstrip(" —-").split())
return text
def _menu_label(kind: str, systems: list[str] | None) -> str:
"""`issue (done) · Plugin & hooks` — the kind, then where it belongs."""
return " · ".join([kind, *systems]) if systems else kind
def _menu_seen_line(note_id: int, kind: str, name: str) -> str:
"""A pointer to a record this session was already shown, not a copy of it."""
return f"> - #{note_id} [{kind} · seen] {name}"
# Max chars of a Process body to fold into the auto-surface description.
_PROC_PREVIEW_CHARS = 200
@@ -1115,8 +1164,9 @@ async def build_autoinject_hint(
lines = [
"> Possibly relevant from your Scribe records — open any in full with "
"`get_note(id)`, or `get_snippet` / `get_process` / `get_lesson` for "
"those kinds (titles only; a line marked `seen` was surfaced earlier "
"this session and may no longer be in context):",
"those kinds. Each line is a record's name, its kind and System, and "
"the passage that matched; a line marked `seen` is a pointer to one "
"already shown this session, so it is in your context:",
]
# THE REGISTER, SAID ONCE AND ONLY WHEN IT APPLIES (milestone 385 step 5).
#
@@ -1152,34 +1202,42 @@ async def build_autoinject_hint(
# with the query that actually matched it.
menu_chunks = _rep_ai.get("best_chunk") or {}
systems = await system_names_for({int(n.id) for _s, n in kept if int(n.id) not in already})
note_ids: list[int] = []
for score, note in kept:
note_ids.append(int(note.id))
title = (note.title or "(untitled)").replace("\n", " ").strip()
line = f"> - #{note.id} [{_record_kind(note)}] \"{title}\" ({score:.2f})"
# ONE WORD, NOT A SENTENCE, and deliberately not the rule arms' phrasing.
# A rule line says "before deciding it does not apply", which is the
# voice of a record that BINDS; a note binds nothing, and borrowing that
# tone would tell the reader a dev-log has authority it does not have.
# The header carries the meaning, so the line carries only the flag.
if int(note.id) in already:
line += " [seen]"
if int(note.id) in stale:
nid = int(note.id)
note_ids.append(nid)
kind = _record_kind(note)
# The NAME, not the title (#4364): a snippet's or lesson's title is its
# embedding shape, trigger and all, and ran past 1,500 characters here.
name = _menu_name(note.title, note.note_type, note.data, note.body)
if nid in already:
# A POINTER, not a copy (#4364). The record is in this session's
# context already — the ledger is cleared at compaction, so "seen"
# stays true — and re-rendering it spent its whole line again for
# nothing. What the reader needs is the reminder that it matched
# again, and the id to open it if it has scrolled out of mind.
line = _menu_seen_line(nid, kind, name)
if nid in stale:
line += " — SUPERSEDED"
lines.append(line)
continue
line = f"> - #{nid} [{_menu_label(kind, systems.get(nid))}] \"{name}\" ({score:.2f})"
if nid in stale:
line += " — SUPERSEDED, a later record covers this; check that first"
if note.user_id != user_id:
who = owners.get(int(note.user_id)) or "another user"
line += f" — shared by {who}, treat as a suggestion"
lines.append(line)
# The passage that earned the line, indented under it. Absent when the
# record has no stored chunk — an un-embedded row, or the reserved
# lesson and reuse slots, which are fetched by their own queries and so
# are not in this search's report. No fallback to the body's opening:
# on a menu that would be a line of preamble dressed as a reason, and a
# reader cannot tell the two apart once they are indented identically.
passage = (menu_chunks.get(int(note.id)) or {}).get("text") or ""
if passage.strip():
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
lines.append(f"> ↳ {short}")
# The passage that earned the line, WHOLE, indented under it (#4364).
# Absent when the record has no stored chunk — an un-embedded row, or
# the reserved lesson and reuse slots, which are fetched by their own
# queries and so are not in this search's report. No fallback to the
# body's opening: on a menu that would be a line of preamble dressed as
# a reason, and a reader cannot tell the two apart once indented alike.
passage = _menu_passage(note.title, (menu_chunks.get(nid) or {}).get("text"), name)
if passage:
lines.append(f"> ↳ {passage}")
# Records what SURVIVED the margin gate, not what the ranker returned — the
# menu the agent actually saw. retrieval_logs already holds the full
@@ -1469,7 +1527,12 @@ def _prior_art_line(item: dict, marker: str, owner: str | None, foreign_lang: st
rather than appended after the title, so the reader sees it while still
reading the score — the two together are the judgement being offered.
"""
title = (item.get("title") or "(untitled)").replace("\n", " ").strip()
# The NAME, not the composed title (#4364) — a snippet's title carries its
# whole trigger and ran to kilobytes on this line, again on every repeat.
title = (
item.get("name") or (item.get("snippet") or {}).get("name")
or (item.get("title") or "(untitled)")
).replace("\n", " ").strip()
mark = f"{marker} · {foreign_lang}" if foreign_lang else marker
line = f"> - #{item['id']} [{mark}] \"{title}\""
if owner:
@@ -2078,6 +2141,7 @@ async def build_write_path_hint(
# dropped — decided by which arm happened to find them — is worse than
# either rule applied consistently: the marker would read as a complete
# account of what the session has met before, and it would not be one.
item["seen"] = nid in excluded
placed.append(("nearby · seen" if nid in excluded else "nearby", item))
# The stamping feed's "actually pulled it" half (#2791). Read once, before
@@ -2255,6 +2319,12 @@ async def build_write_path_hint(
marker,
{
"id": int(note.id), "title": note.title, "user_id": note.user_id,
# The name the line shows, and whether this session has
# it already — carried as data for the reason `kind` is
# (#4364): the line is built from facts, not from
# re-reading its own marker.
"name": _menu_name(note.title, note.note_type, note.data, note.body),
"seen": int(note.id) in excluded,
# Carried, not re-read off the rendered marker. The
# marker is prose assembled for a human and it already
# varies by kind, language and the `seen` flag — a
@@ -2399,8 +2469,9 @@ async def build_write_path_hint(
"`get_lesson(id)` for a lesson, `get_note(id)` otherwise. Reuse a "
"snippet rather than writing a fresh one-off; read an issue before "
"repeating what it records "
"(titles only; a line marked `seen` was surfaced earlier this "
"session and may no longer be in context):"
"(each line is a record's name and kind, with the passage that "
"matched; a line marked `seen` is a pointer to one already shown "
"this session, so it is in your context):"
)
# The same clause the prompt menu carries, on the same condition and for
# the same reason: this menu's three other kinds are all things that WERE
@@ -2434,10 +2505,16 @@ async def build_write_path_hint(
# is no matching passage and the body's opening would be a fabricated
# reason. Absence here is meaningful: a line with no passage under it is
# one that earned its place by where it lives, not by what it says.
passage = (wp_chunks.get(int(item["id"])) or {}).get("text") or ""
if passage.strip():
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
lines.append(f"> ↳ {short}")
# WHOLE, and only on first sight (#4364): a `seen` line is a pointer to
# a record already in context, and its passage is already there too.
if item.get("seen"):
continue
passage = _menu_passage(
item.get("title"), (wp_chunks.get(int(item["id"])) or {}).get("text"),
item.get("name") or "",
)
if passage:
lines.append(f"> ↳ {passage}")
if stamped:
lines.append(_stamp_line(path, stamped))