fix(telemetry): the human half of the pull ledger recorded one kind in three
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 20s
CI & Build / Python tests (push) Successful in 47s
CI & Build / Build & push image (push) Successful in 27s

Opening a snippet in the UI recorded rest_snippet. Opening a note or a task
recorded nothing — so the most direct evidence the product has that anyone
cares about a record existed for one kind out of three, and the other two sat
at zero pulls looking like dead weight beside a kind that merely had a counter.

Not an open question about intent: models/note_usage.py already documented
'rest_note' as a source value. Nothing wrote it. The design named it and the
implementation stopped at snippets.

Adds rest_note and rest_task. Tagged by SURFACE rather than by the record's
kind, matching rest_snippet — the kind is a join away, but which surface asked
is not recoverable after the fact. The mcp_/rest_ split stays load-bearing:
"is this dead weight" is served by any pull, "was that injected line useful" by
agent pulls alone, and a human clicking a link would inflate exactly the number
#1038 and #2085 gate on.

The vocabulary comment in the model was itself the stale-enumeration shape this
survey keeps finding — it named a source nothing wrote while omitting sources
that existed. Replaced with the naming CONVENTION plus a pointer to grep, which
cannot drift, rather than a longer list that would go stale the same way.

Guard extended to the REST surface, same derivation as the MCP half: a route
registered at exactly /<int:x> for GET is a detail view, and one reaching a
note-backed loader must record. Handler source is expanded one level through
module-private helpers, without which get_snippet_route — the route that
already got this right — would drop out of the check by loading via
_load_snippet. Verified the guard fires when a call is removed.

Renamed test_mcp_pull_telemetry.py -> test_pull_telemetry.py; it is no longer
only about MCP.

Closes #2476
This commit is contained in:
2026-08-06 08:51:21 -04:00
parent ac1ce0a7f0
commit c18139622c
4 changed files with 122 additions and 7 deletions
+203
View File
@@ -0,0 +1,203 @@
"""Every surface that opens ONE note-backed record must record the pull.
Covers both halves: the MCP getters an agent calls, and the REST detail views a
human opens. They are one ledger with two prefixes (`mcp_*` / `rest_*`), and a
gap on either side makes the same records look untouched.
WHY THIS EXISTS
`note_usage_events` answers "did anyone ever actually open this?" — the
surfaced:pulled ratio is what makes dead weight visible and prunable. A getter
that opens a record without recording it leaves that kind permanently at zero
pulls, so it looks like dead weight beside kinds that merely had a counter.
That has now happened twice:
#2245 `get_task` recorded nothing while auto-inject surfaced mostly tasks.
Fixed by adding the call to notes, tasks and snippets.
#2476 `get_process` recorded nothing — and the auto-inject menu header names
`get_process` as the way to open that kind. Processes were embedded
when #2245 was fixed; the fix enumerated the kinds someone thought of
rather than the kinds that exist. The same issue found the REST half:
`rest_snippet` was recorded, note and task detail were not, and the
model's own comment named a `'rest_note'` source nothing wrote.
A missing call is the shape no behavioural test catches: it changes no return
value (#2278, shape 4). Source inspection is the only thing that sees it.
WHAT MAKES THIS DERIVED RATHER THAN A LIST
The getters are not enumerated here. They are discovered from the tool modules
by AST, and the ones that must record are identified by the loader they call —
so a `get_<newkind>` added tomorrow is covered the moment it loads a note the
way every other getter does.
The loader names ARE a list, and that is the residual weakness. The second test
pins them against a RENAME — the failure mode that would silently empty the
candidate set and let this pass while checking nothing.
It does not discover NEW loaders, and an earlier draft that tried to failed for
the wrong reason: `create_note` and `update_note` also return a `Note`, so an
annotation scan finds writers, not readers. Distinguishing them needs more than
a type, so the honest position is a pinned list plus a non-empty assertion,
and this paragraph saying so.
"""
from __future__ import annotations
import ast
import inspect
import pathlib
import pkgutil
# Loaders that return ONE note-backed record in full. A getter calling any of
# these is opening a record, which is the act `pulled` describes.
#
# `list_notes` is deliberately absent: `get_milestone` calls it to list a
# milestone's steps, and that is a LIST — the milestone itself is not a note,
# and its steps are surfaced rather than opened.
SINGLE_NOTE_LOADERS = (
"get_note_for_user",
"resolve_process",
"get_snippet",
)
_SRC = pathlib.Path(__file__).resolve().parents[1] / "src" / "scribe"
TOOLS_DIR = _SRC / "mcp" / "tools"
ROUTES_DIR = _SRC / "routes"
def _getters():
"""(module name, function name, source) for every `get_*` MCP tool."""
for mod in pkgutil.iter_modules([str(TOOLS_DIR)]):
path = TOOLS_DIR / f"{mod.name}.py"
source = path.read_text()
for node in ast.parse(source).body:
if isinstance(node, ast.AsyncFunctionDef) and node.name.startswith("get_"):
yield mod.name, node.name, ast.get_source_segment(source, node) or ""
def test_every_single_record_getter_records_a_pull():
missing = []
checked = []
for module, name, body in _getters():
if not any(loader in body for loader in SINGLE_NOTE_LOADERS):
continue
checked.append(f"{module}.{name}")
if "record_pulled" not in body:
missing.append(f"{module}.{name}")
# If this ever drops to zero the test has stopped testing anything — a
# renamed loader would silently empty the candidate set and pass.
assert checked, "found no note-backed getters; the loader names must have moved"
assert not missing, (
f"these getters open a record without recording the pull: {missing}. "
f"Add record_pulled(user_id=…, note_id=…, source='mcp_<tool>') before "
f"returning — see mcp/tools/notes.py:get_note."
)
def _detail_routes():
"""(module, handler, expanded source) for every bare-id GET route.
A route registered at exactly `/<int:x>` for GET is the DETAIL view of one
record — that shape is what distinguishes it from a list, a sub-resource
(`/<int:x>/versions`) or a write. Nothing else about the handler has to be
guessed.
The source is expanded one level through module-private helpers, because
`get_snippet_route` loads via `_load_snippet` rather than calling the loader
itself. Without the expansion the snippet route — the one that already got
this right — would drop out of the check.
"""
import re
bare_id = re.compile(r"^/<int:\w+>$")
for path in sorted(ROUTES_DIR.glob("*.py")):
source = path.read_text()
tree = ast.parse(source)
helpers = {
node.name: ast.get_source_segment(source, node) or ""
for node in tree.body
if isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef))
and node.name.startswith("_")
}
for node in tree.body:
if not isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef)):
continue
for dec in node.decorator_list:
if not isinstance(dec, ast.Call):
continue
route = next((a.value for a in dec.args
if isinstance(a, ast.Constant)), None)
if not isinstance(route, str) or not bare_id.match(route):
continue
methods = [
e.value for kw in dec.keywords if kw.arg == "methods"
and isinstance(kw.value, ast.List)
for e in kw.value.elts if isinstance(e, ast.Constant)
]
if "GET" not in methods:
continue
body = ast.get_source_segment(source, node) or ""
expanded = body + "".join(
src for name, src in helpers.items() if name in body
)
yield path.name, node.name, expanded
def test_every_rest_detail_view_records_a_pull():
"""The human half of the same ledger.
`rest_snippet` was recorded; note and task detail recorded nothing, so the
UI's most direct evidence of interest — someone opened the record — existed
for one kind out of three. The model's own comment listed `'rest_note'` as
a source, which means the design intended it and the implementation stopped
at snippets.
Same derivation as the MCP test above, over the other surface: the routes
are discovered, and the ones that must record are identified by the loader
they reach. A `/<int:x>` GET added for a fourth note-backed kind is covered
the day it is written.
"""
missing = []
checked = []
for module, handler, body in _detail_routes():
if not any(loader in body for loader in SINGLE_NOTE_LOADERS):
continue # not note-backed — groups and projects land here
checked.append(f"{module}:{handler}")
if "record_pulled" not in body:
missing.append(f"{module}:{handler}")
assert checked, (
"found no note-backed detail routes; the route shape or the loader "
"names must have moved"
)
assert not missing, (
f"these detail views open a record without recording the pull: "
f"{missing}. Add record_pulled(user_id=…, note_id=…, source='rest_<kind>') "
f"before returning — see routes/snippets.py:get_snippet_route."
)
def test_every_named_loader_still_exists():
"""Pins the hand-written list against a rename.
A renamed loader is the failure that matters: the candidate set above would
quietly empty and the first test would pass while checking nothing. The
`assert checked` there catches it too; this says WHICH name moved, which is
the difference between a five-minute fix and a puzzle.
"""
from scribe.services import notes as notes_svc
from scribe.services import snippets as snippets_svc
available = {
name
for svc in (notes_svc, snippets_svc)
for name, obj in vars(svc).items()
if inspect.iscoroutinefunction(obj)
}
gone = [name for name in SINGLE_NOTE_LOADERS if name not in available]
assert not gone, (
f"SINGLE_NOTE_LOADERS names {gone} that no longer exist — they were "
f"renamed or moved. Update the list, or the pull check silently stops "
f"covering whatever used them."
)