bvandeusenandClaude Opus 5
2b85443dd1
CI & Build / Python lint (push) Successful in 2s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 34s
CI & Build / Python tests (push) Successful in 56s
CI & Build / Build & push image (push) Successful in 44s
retrieval_logs answers "what did the ranker return, at what scores" — the right substrate for tuning a threshold. It cannot answer the question the snippet corpus actually needs: did anyone open this? A snippet nobody opens is not neutral. It takes a slot in every future auto-inject menu and crowds out something useful. Adds note_usage_events (migration 0071): one row per note per event, either 'surfaced' (we put its title in front of an agent) or 'pulled' (someone opened it in full), tagged with which surface produced it. Closes the gap #2082 recorded against this work. The write-path PLACE arm carries no score, so it has no home in retrieval_logs — folding it in would corrupt the score distribution that table exists to capture. The result was that the arm firing on the STRONGEST claim ("there is already a canonical helper in this exact file") was the one arm nobody could measure. Both arms now emit usage events under distinct sources, so their pull-through rates are finally comparable. Deliberate departures from the task as written: - Not in-session correlation. The original framing was "correlate result_ids against a later get_note in the same session." There is no session identity server-side — the MCP endpoint is stateless and the hooks send no session id — and adding one would mean threading an opaque client-supplied token through every read path. Two independent counters answer the question without it: surfaced 40×, pulled 0 is dead weight regardless of how those events distribute across sessions. - Pulls record at the ENTRY POINTS (MCP tools, REST detail route), not in snippets_svc.get_snippet, which update and merge also reach. Counting those would inflate precisely the number meant to say "someone chose to look at this." - get_note records for every note kind, not just snippets. The auto-inject menu surfaces tasks and processes too; scoping this to snippets would pin those at zero pulls forever and make them read as dead weight next to snippets that merely had a counter. Surfaced in the Snippets list as an "N/M used" badge, warning-toned once a record has been offered 3+ times and never opened, with the tooltip saying what to do about it (usually: its "when to reach for it" doesn't say when). No badge at all below one surfacing — "0/0" reads as a verdict when it's an absence of evidence. Also returned from MCP list_snippets so the agent can see dead weight without opening the UI. Telemetry keeps the retrieval_telemetry contract throughout: writes are fire-and-forget, reads degrade to zeroes, and no path can raise into the surface it observes. Refs #2085 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
77 lines
3.5 KiB
Python
77 lines
3.5 KiB
Python
from datetime import datetime, timezone
|
|
|
|
from sqlalchemy import DateTime, Index, Integer, Text
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from scribe.models import Base
|
|
|
|
SURFACED = "surfaced"
|
|
PULLED = "pulled"
|
|
|
|
|
|
class NoteUsageEvent(Base):
|
|
"""One row per time a note was SURFACED to the agent, or PULLED in full.
|
|
|
|
Answers the question RetrievalLog cannot: not "what did the ranker return
|
|
and with what scores" but "did anyone ever actually open this?" A snippet
|
|
nobody opens is not neutral — it competes for the injection budget on every
|
|
future turn and dilutes the menu — so the surfaced:pulled ratio is what
|
|
makes dead weight visible and prunable.
|
|
|
|
WHY A SEPARATE TABLE FROM RetrievalLog. RetrievalLog is one row per *call*,
|
|
keyed on the score distribution it exists to capture; folding un-scored
|
|
events into it would corrupt exactly the distribution threshold tuning reads
|
|
(see the KNOWN GAP note this closes in plugin_context.build_write_path_hint).
|
|
This table is one row per *note per event*, which is the grain the usage
|
|
readout needs and the grain RetrievalLog's JSONB `result_ids` can't be
|
|
indexed at. The two are complements: RetrievalLog tunes the threshold, this
|
|
tunes the corpus.
|
|
|
|
WHY NOT IN-SESSION CORRELATION. The original framing was "correlate
|
|
result_ids against a later get_note in the same session". There is no
|
|
session identity server-side — the MCP endpoint is stateless and hooks send
|
|
no session id — and introducing one would mean threading an opaque,
|
|
client-supplied token through every read path for a signal that does not
|
|
need it. Two independent counters answer the question without it: a note
|
|
surfaced 40 times and never pulled is dead weight regardless of which
|
|
sessions those events fell in.
|
|
|
|
Deliberately FK-free on user_id and note_id (mirrors RetrievalLog/AppLog):
|
|
telemetry outlives the row it describes, and deleting a note should not
|
|
erase the evidence that it was surfaced 40 times and never once opened.
|
|
"""
|
|
|
|
__tablename__ = "note_usage_events"
|
|
|
|
id: Mapped[int] = mapped_column(primary_key=True)
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
|
|
)
|
|
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
note_id: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
# 'surfaced' | 'pulled'
|
|
event: Mapped[str] = mapped_column(Text, nullable=False)
|
|
# Which surface produced it: 'auto_inject' | 'write_path_place' |
|
|
# 'write_path_semantic' | 'mcp_get_snippet' | 'mcp_get_note' | 'rest_note'.
|
|
# Kept granular so the place arm and the semantic arm can be compared —
|
|
# that comparison is the whole reason the place arm needed logging at all.
|
|
source: Mapped[str] = mapped_column(Text, nullable=False)
|
|
|
|
__table_args__ = (
|
|
# The readout is always "these note ids, split by event" — a covering
|
|
# composite beats separate single-column indexes for it.
|
|
Index("ix_note_usage_note_event", "note_id", "event"),
|
|
Index("ix_note_usage_created_at", "created_at"),
|
|
Index("ix_note_usage_user_id", "user_id"),
|
|
)
|
|
|
|
def to_dict(self) -> dict:
|
|
return {
|
|
"id": self.id,
|
|
"created_at": self.created_at.isoformat() if self.created_at else None,
|
|
"user_id": self.user_id,
|
|
"note_id": self.note_id,
|
|
"event": self.event,
|
|
"source": self.source,
|
|
}
|