feat(retrieval): a menu line is a name, its kind and System, and the whole passage that matched (#4364)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 58s
CI & Build / Python tests (push) Successful in 1m36s
CI & Build / Build & push image (push) Successful in 26s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 58s
CI & Build / Python tests (push) Successful in 1m36s
CI & Build / Build & push image (push) Successful in 26s
Injected lines rendered a snippet's or lesson's title, which carries its whole trigger by construction (the embedding shape) and ran past 1,500 characters -- again on every `seen` repeat. The passage under a line was cut to 200 chars from the middle, keeping its head (the title again) and losing where the match was. Now, on both the prompt menu and the write-path prior-art menu: - the line shows the record's NAME (snippet data.name / lesson subject), with its kind and System (`[issue (done) · Plugin & hooks]`); - the passage is the whole matched chunk, title prefix stripped, on one line so the blockquote holds; a title-only match hands over the trigger; - a `seen` record is a one-line pointer to what is already in context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -40,6 +40,7 @@ from scribe.services.retrieval_surfaces import (
|
||||
)
|
||||
from scribe.services.retrieval_telemetry import record_retrieval
|
||||
from scribe.services.settings import get_setting
|
||||
from scribe.services.systems import system_names_for
|
||||
from scribe.services.text import elide
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -47,23 +48,71 @@ logger = logging.getLogger(__name__)
|
||||
# Defensive cap below Claude Code's 10k additionalContext limit.
|
||||
_MAX_CHARS = 9000
|
||||
|
||||
# Max chars of the matched passage shown under an injected menu line.
|
||||
# WHAT A MENU LINE CARRIES (#4364): the record's NAME, its kind and System,
|
||||
# and the WHOLE passage that matched. Metadata plus the evidence, rather than a
|
||||
# title asked to be both.
|
||||
#
|
||||
# The menu used to be titles alone, on the reasoning that its job is AWARENESS —
|
||||
# make the agent know the record exists and reach for it, not dump it. That
|
||||
# holds for a lesson or a snippet, whose title carries its trigger by
|
||||
# construction ("what — when it applies"). It does not hold for an issue, a
|
||||
# dev-log or a plain note, where the title is a headline and the reason this
|
||||
# record matched is a sentence somewhere inside it. The reader was being asked
|
||||
# "is this worth opening?" and handed the one part of the record guaranteed not
|
||||
# to answer it.
|
||||
# The name, not the title. A snippet's or lesson's title is `name — when it
|
||||
# applies` by construction (`embeddings.trigger_title`), because that join is
|
||||
# what makes it rank on its situation. That is an EMBEDDING shape, and rendered
|
||||
# as a menu line it ran to 1,500+ characters — the trigger paragraph spent
|
||||
# again on every line, and again on every repeat. The trigger still arrives
|
||||
# when it is what matched: it is in the chunk, and `_menu_passage` hands it
|
||||
# over when the title was the whole match.
|
||||
#
|
||||
# 200 rather than more because this is a menu: eight lines at 200 is ~1.6KB,
|
||||
# which buys the decision without turning an awareness push into a dump. It is
|
||||
# the PASSAGE THAT MATCHED, not the record's opening — the search already knows
|
||||
# which one that is and used to throw it away (#4243) — so 200 characters here
|
||||
# are worth far more than 200 characters of preamble.
|
||||
_MENU_PASSAGE_CHARS = 200
|
||||
# The whole passage, not 200 characters of it. The search already chose the
|
||||
# chunk that matched; the old cut kept its head and tail, and the head is the
|
||||
# title every chunk is prefixed with — so the reader got the title twice and
|
||||
# lost the middle, which is where the match was (lesson #4248). A chunk is at
|
||||
# most ~1.4 KB (`embeddings._CHUNK_CHAR_BUDGET`), and it is shown once: a
|
||||
# repeat is a one-line pointer (`_menu_seen_line`), not a second copy.
|
||||
|
||||
|
||||
def _menu_name(title: str | None, note_type: str | None, data=None, body: str | None = "") -> str:
|
||||
"""The record's name — its title without the trigger composed into it."""
|
||||
title = (title or "(untitled)").replace("\n", " ").strip()
|
||||
data = data if isinstance(data, dict) else {}
|
||||
if note_type == "snippet":
|
||||
from scribe.services.embeddings import TRIGGER_SEP
|
||||
return (data.get("name") or title.partition(TRIGGER_SEP)[0]).strip() or title
|
||||
if note_type == LESSON_NOTE_TYPE:
|
||||
from types import SimpleNamespace
|
||||
|
||||
from scribe.services.embeddings import untrigger_title
|
||||
from scribe.services.lessons import lesson_trigger
|
||||
trigger = lesson_trigger(SimpleNamespace(data=data, body=body or ""))
|
||||
return untrigger_title(title, trigger).strip() or title
|
||||
return title
|
||||
|
||||
|
||||
def _menu_passage(title: str | None, chunk_text: str | None, name: str = "") -> str:
|
||||
"""The matched chunk on one line, without the title it was embedded under.
|
||||
|
||||
Every chunk is `title\nsection` (`embeddings.embedding_text`), so the title
|
||||
prefix is stripped exactly. A chunk that WAS only the title — a short
|
||||
record, or the head chunk of one — matched on the title, and for a
|
||||
trigger-keyed kind the part of it the name line no longer shows is the
|
||||
trigger: that is returned, because it is precisely what matched.
|
||||
One line, so the menu's blockquote survives it.
|
||||
"""
|
||||
title = (title or "").strip()
|
||||
text = (chunk_text or "").strip()
|
||||
if title and text.startswith(title):
|
||||
text = text[len(title):]
|
||||
text = " ".join(text.split())
|
||||
if not text and name and title.startswith(name) and title != name:
|
||||
text = " ".join(title[len(name):].lstrip(" —-").split())
|
||||
return text
|
||||
|
||||
|
||||
def _menu_label(kind: str, systems: list[str] | None) -> str:
|
||||
"""`issue (done) · Plugin & hooks` — the kind, then where it belongs."""
|
||||
return " · ".join([kind, *systems]) if systems else kind
|
||||
|
||||
|
||||
def _menu_seen_line(note_id: int, kind: str, name: str) -> str:
|
||||
"""A pointer to a record this session was already shown, not a copy of it."""
|
||||
return f"> - #{note_id} [{kind} · seen] {name}"
|
||||
|
||||
# Max chars of a Process body to fold into the auto-surface description.
|
||||
_PROC_PREVIEW_CHARS = 200
|
||||
@@ -1115,8 +1164,9 @@ async def build_autoinject_hint(
|
||||
lines = [
|
||||
"> Possibly relevant from your Scribe records — open any in full with "
|
||||
"`get_note(id)`, or `get_snippet` / `get_process` / `get_lesson` for "
|
||||
"those kinds (titles only; a line marked `seen` was surfaced earlier "
|
||||
"this session and may no longer be in context):",
|
||||
"those kinds. Each line is a record's name, its kind and System, and "
|
||||
"the passage that matched; a line marked `seen` is a pointer to one "
|
||||
"already shown this session, so it is in your context:",
|
||||
]
|
||||
# THE REGISTER, SAID ONCE AND ONLY WHEN IT APPLIES (milestone 385 step 5).
|
||||
#
|
||||
@@ -1152,34 +1202,42 @@ async def build_autoinject_hint(
|
||||
# with the query that actually matched it.
|
||||
menu_chunks = _rep_ai.get("best_chunk") or {}
|
||||
|
||||
systems = await system_names_for({int(n.id) for _s, n in kept if int(n.id) not in already})
|
||||
note_ids: list[int] = []
|
||||
for score, note in kept:
|
||||
note_ids.append(int(note.id))
|
||||
title = (note.title or "(untitled)").replace("\n", " ").strip()
|
||||
line = f"> - #{note.id} [{_record_kind(note)}] \"{title}\" ({score:.2f})"
|
||||
# ONE WORD, NOT A SENTENCE, and deliberately not the rule arms' phrasing.
|
||||
# A rule line says "before deciding it does not apply", which is the
|
||||
# voice of a record that BINDS; a note binds nothing, and borrowing that
|
||||
# tone would tell the reader a dev-log has authority it does not have.
|
||||
# The header carries the meaning, so the line carries only the flag.
|
||||
if int(note.id) in already:
|
||||
line += " [seen]"
|
||||
if int(note.id) in stale:
|
||||
nid = int(note.id)
|
||||
note_ids.append(nid)
|
||||
kind = _record_kind(note)
|
||||
# The NAME, not the title (#4364): a snippet's or lesson's title is its
|
||||
# embedding shape, trigger and all, and ran past 1,500 characters here.
|
||||
name = _menu_name(note.title, note.note_type, note.data, note.body)
|
||||
if nid in already:
|
||||
# A POINTER, not a copy (#4364). The record is in this session's
|
||||
# context already — the ledger is cleared at compaction, so "seen"
|
||||
# stays true — and re-rendering it spent its whole line again for
|
||||
# nothing. What the reader needs is the reminder that it matched
|
||||
# again, and the id to open it if it has scrolled out of mind.
|
||||
line = _menu_seen_line(nid, kind, name)
|
||||
if nid in stale:
|
||||
line += " — SUPERSEDED"
|
||||
lines.append(line)
|
||||
continue
|
||||
line = f"> - #{nid} [{_menu_label(kind, systems.get(nid))}] \"{name}\" ({score:.2f})"
|
||||
if nid in stale:
|
||||
line += " — SUPERSEDED, a later record covers this; check that first"
|
||||
if note.user_id != user_id:
|
||||
who = owners.get(int(note.user_id)) or "another user"
|
||||
line += f" — shared by {who}, treat as a suggestion"
|
||||
lines.append(line)
|
||||
# The passage that earned the line, indented under it. Absent when the
|
||||
# record has no stored chunk — an un-embedded row, or the reserved
|
||||
# lesson and reuse slots, which are fetched by their own queries and so
|
||||
# are not in this search's report. No fallback to the body's opening:
|
||||
# on a menu that would be a line of preamble dressed as a reason, and a
|
||||
# reader cannot tell the two apart once they are indented identically.
|
||||
passage = (menu_chunks.get(int(note.id)) or {}).get("text") or ""
|
||||
if passage.strip():
|
||||
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
|
||||
lines.append(f"> ↳ {short}")
|
||||
# The passage that earned the line, WHOLE, indented under it (#4364).
|
||||
# Absent when the record has no stored chunk — an un-embedded row, or
|
||||
# the reserved lesson and reuse slots, which are fetched by their own
|
||||
# queries and so are not in this search's report. No fallback to the
|
||||
# body's opening: on a menu that would be a line of preamble dressed as
|
||||
# a reason, and a reader cannot tell the two apart once indented alike.
|
||||
passage = _menu_passage(note.title, (menu_chunks.get(nid) or {}).get("text"), name)
|
||||
if passage:
|
||||
lines.append(f"> ↳ {passage}")
|
||||
|
||||
# Records what SURVIVED the margin gate, not what the ranker returned — the
|
||||
# menu the agent actually saw. retrieval_logs already holds the full
|
||||
@@ -1469,7 +1527,12 @@ def _prior_art_line(item: dict, marker: str, owner: str | None, foreign_lang: st
|
||||
rather than appended after the title, so the reader sees it while still
|
||||
reading the score — the two together are the judgement being offered.
|
||||
"""
|
||||
title = (item.get("title") or "(untitled)").replace("\n", " ").strip()
|
||||
# The NAME, not the composed title (#4364) — a snippet's title carries its
|
||||
# whole trigger and ran to kilobytes on this line, again on every repeat.
|
||||
title = (
|
||||
item.get("name") or (item.get("snippet") or {}).get("name")
|
||||
or (item.get("title") or "(untitled)")
|
||||
).replace("\n", " ").strip()
|
||||
mark = f"{marker} · {foreign_lang}" if foreign_lang else marker
|
||||
line = f"> - #{item['id']} [{mark}] \"{title}\""
|
||||
if owner:
|
||||
@@ -2078,6 +2141,7 @@ async def build_write_path_hint(
|
||||
# dropped — decided by which arm happened to find them — is worse than
|
||||
# either rule applied consistently: the marker would read as a complete
|
||||
# account of what the session has met before, and it would not be one.
|
||||
item["seen"] = nid in excluded
|
||||
placed.append(("nearby · seen" if nid in excluded else "nearby", item))
|
||||
|
||||
# The stamping feed's "actually pulled it" half (#2791). Read once, before
|
||||
@@ -2255,6 +2319,12 @@ async def build_write_path_hint(
|
||||
marker,
|
||||
{
|
||||
"id": int(note.id), "title": note.title, "user_id": note.user_id,
|
||||
# The name the line shows, and whether this session has
|
||||
# it already — carried as data for the reason `kind` is
|
||||
# (#4364): the line is built from facts, not from
|
||||
# re-reading its own marker.
|
||||
"name": _menu_name(note.title, note.note_type, note.data, note.body),
|
||||
"seen": int(note.id) in excluded,
|
||||
# Carried, not re-read off the rendered marker. The
|
||||
# marker is prose assembled for a human and it already
|
||||
# varies by kind, language and the `seen` flag — a
|
||||
@@ -2399,8 +2469,9 @@ async def build_write_path_hint(
|
||||
"`get_lesson(id)` for a lesson, `get_note(id)` otherwise. Reuse a "
|
||||
"snippet rather than writing a fresh one-off; read an issue before "
|
||||
"repeating what it records "
|
||||
"(titles only; a line marked `seen` was surfaced earlier this "
|
||||
"session and may no longer be in context):"
|
||||
"(each line is a record's name and kind, with the passage that "
|
||||
"matched; a line marked `seen` is a pointer to one already shown "
|
||||
"this session, so it is in your context):"
|
||||
)
|
||||
# The same clause the prompt menu carries, on the same condition and for
|
||||
# the same reason: this menu's three other kinds are all things that WERE
|
||||
@@ -2434,10 +2505,16 @@ async def build_write_path_hint(
|
||||
# is no matching passage and the body's opening would be a fabricated
|
||||
# reason. Absence here is meaningful: a line with no passage under it is
|
||||
# one that earned its place by where it lives, not by what it says.
|
||||
passage = (wp_chunks.get(int(item["id"])) or {}).get("text") or ""
|
||||
if passage.strip():
|
||||
short, _cut = elide(" ".join(passage.split()), _MENU_PASSAGE_CHARS)
|
||||
lines.append(f"> ↳ {short}")
|
||||
# WHOLE, and only on first sight (#4364): a `seen` line is a pointer to
|
||||
# a record already in context, and its passage is already there too.
|
||||
if item.get("seen"):
|
||||
continue
|
||||
passage = _menu_passage(
|
||||
item.get("title"), (wp_chunks.get(int(item["id"])) or {}).get("text"),
|
||||
item.get("name") or "",
|
||||
)
|
||||
if passage:
|
||||
lines.append(f"> ↳ {passage}")
|
||||
|
||||
if stamped:
|
||||
lines.append(_stamp_line(path, stamped))
|
||||
|
||||
Reference in New Issue
Block a user