feat(snippets): usage signal — was a surfaced record ever actually pulled?
CI & Build / Python lint (push) Successful in 2s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 34s
CI & Build / Python tests (push) Successful in 56s
CI & Build / Build & push image (push) Successful in 44s

retrieval_logs answers "what did the ranker return, at what scores" — the
right substrate for tuning a threshold. It cannot answer the question the
snippet corpus actually needs: did anyone open this? A snippet nobody
opens is not neutral. It takes a slot in every future auto-inject menu and
crowds out something useful.

Adds note_usage_events (migration 0071): one row per note per event,
either 'surfaced' (we put its title in front of an agent) or 'pulled'
(someone opened it in full), tagged with which surface produced it.

Closes the gap #2082 recorded against this work. The write-path PLACE arm
carries no score, so it has no home in retrieval_logs — folding it in
would corrupt the score distribution that table exists to capture. The
result was that the arm firing on the STRONGEST claim ("there is already a
canonical helper in this exact file") was the one arm nobody could
measure. Both arms now emit usage events under distinct sources, so their
pull-through rates are finally comparable.

Deliberate departures from the task as written:

- Not in-session correlation. The original framing was "correlate
  result_ids against a later get_note in the same session." There is no
  session identity server-side — the MCP endpoint is stateless and the
  hooks send no session id — and adding one would mean threading an
  opaque client-supplied token through every read path. Two independent
  counters answer the question without it: surfaced 40×, pulled 0 is dead
  weight regardless of how those events distribute across sessions.

- Pulls record at the ENTRY POINTS (MCP tools, REST detail route), not in
  snippets_svc.get_snippet, which update and merge also reach. Counting
  those would inflate precisely the number meant to say "someone chose to
  look at this."

- get_note records for every note kind, not just snippets. The auto-inject
  menu surfaces tasks and processes too; scoping this to snippets would
  pin those at zero pulls forever and make them read as dead weight next
  to snippets that merely had a counter.

Surfaced in the Snippets list as an "N/M used" badge, warning-toned once a
record has been offered 3+ times and never opened, with the tooltip saying
what to do about it (usually: its "when to reach for it" doesn't say
when). No badge at all below one surfacing — "0/0" reads as a verdict when
it's an absence of evidence. Also returned from MCP list_snippets so the
agent can see dead weight without opening the UI.

Telemetry keeps the retrieval_telemetry contract throughout: writes are
fire-and-forget, reads degrade to zeroes, and no path can raise into the
surface it observes.

Refs #2085

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-28 18:09:17 -04:00
parent c569cdd0eb
commit 2b85443dd1
11 changed files with 626 additions and 12 deletions
+25 -6
View File
@@ -29,6 +29,7 @@ from scribe.services import rulebooks as rulebooks_svc
from scribe.services import snippets as snippets_svc
from scribe.services.access import label_shared_items, owner_names_for
from scribe.services.embeddings import semantic_search_notes
from scribe.services.note_usage import record_surfaced
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.settings import get_setting
@@ -271,6 +272,12 @@ async def build_autoinject_hint(
line += f" — shared by {who}, treat as a suggestion"
lines.append(line)
# Records what SURVIVED the margin gate, not what the ranker returned — the
# menu the agent actually saw. retrieval_logs already holds the full
# candidate set for threshold tuning; conflating the two would make
# "surfaced" mean two different things depending on the surface (#2085).
record_surfaced(user_id=user_id, note_ids=note_ids, source="auto_inject")
return {"context": "\n".join(lines), "note_ids": note_ids, "config": cfg}
@@ -341,12 +348,12 @@ async def build_write_path_hint(
arm is logged to retrieval_logs as source='write_path' — its own source, so
its precision is tunable separately from auto-inject's.
KNOWN GAP: only the semantic arm is logged, matching auto-inject's convention
of recording the candidate set for threshold tuning. Location hits carry no
score, so folding them in would corrupt the score distribution the log exists
to capture — but it does mean a snippet surfaced BY PLACE leaves no trace. The
usage signal (#2085) correlates retrieval_logs against later pulls, so it will
need a home for un-scored surfacing before it can measure this arm.
Location hits still carry no score and so stay out of retrieval_logs, whose
score distribution they would corrupt. What closed the gap (#2085) is that
un-scored surfacing now has its own home: BOTH arms emit note_usage_events,
tagged 'write_path_place' vs 'write_path_semantic', so the place arm is
finally measurable — and the two arms' pull-through rates are comparable,
which is the number that says whether place really does beat meaning here.
"""
cfg = await get_writepath_config(user_id)
empty = {"context": "", "note_ids": [], "config": cfg}
@@ -439,6 +446,18 @@ async def build_write_path_hint(
owner = owners.get(int(owner_id)) or "another user"
lines.append(_prior_art_line(item, marker, owner))
# Split by arm, which is the whole reason this table exists. The place arm
# carries no score and so has no home in retrieval_logs; before #2085 a
# snippet surfaced BY PLACE left no trace anywhere, making the arm that
# fires on the strongest possible claim ("there is already a canonical
# helper in this exact file") the one arm nobody could measure.
by_arm: dict[str, list[int]] = {}
for marker, item in menu:
arm = "write_path_place" if marker in ("here", "nearby") else "write_path_semantic"
by_arm.setdefault(arm, []).append(int(item["id"]))
for arm, ids in by_arm.items():
record_surfaced(user_id=user_id, note_ids=ids, source=arm)
return {"context": "\n".join(lines), "note_ids": note_ids, "config": cfg}