fix(telemetry): record a pull on get_task, closing the surfaced→pulled loop
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 10s
CI & Build / integration (push) Successful in 12s
CI & Build / Python tests (push) Successful in 43s
CI & Build / Build & push image (push) Successful in 39s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 10s
CI & Build / integration (push) Successful in 12s
CI & Build / Python tests (push) Successful in 43s
CI & Build / Build & push image (push) Successful in 39s
Closes #2245. note_usage_events recorded `surfaced` for every auto-inject menu line regardless of kind, but `pulled` only from get_note, get_snippet and the REST snippet route. get_task recorded nothing. Auto-inject ranks kind-blind over a corpus that is overwhelmingly tasks and issues, so tasks are most of what it surfaces. Measured live, "write a function to debounce a callback in the frontend" returned three tasks and zero snippets — all three written as surfaced, none able to record a pull. surfaced and pulled only mean anything as a PAIR; the rate between them is what #1038 and #2085 gate on. So the gap sat exactly where the volume is, and the metric would have said "auto-inject surfaces things nobody opens" for its own dominant kind — an artifact of the instrumentation, not a fact about the feature, and one that pointed at a plausible-sounding wrong conclusion. get_note already carried a comment stating this was meant to cover ANY note kind precisely so tasks wouldn't look like dead weight. get_task is a separate tool in a separate module and never got the call — sibling drift, invisible because a missing side effect changes no return value. Guarded by a rule-#33 contract test that asserts, by source inspection, that every getter reachable from an auto-inject menu calls record_pulled. Source inspection because no behavioural test can see a call that isn't there. Not fixed here: the REST note/task detail routes still record nothing while the REST snippet route records `rest_snippet`. That asymmetry is real, but a human reading a note in a browser is arguably not the same event as an agent recalling one, and collapsing them could skew the signal the other way. Raised as a question for the retrieval survey instead of decided in passing. Pre-fix rows under-count task pulls, one-sidedly by kind — treat them as unknown rather than zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -27,6 +27,7 @@ from scribe.services import rulebooks as rulebooks_svc
|
||||
from scribe.services import systems as systems_svc
|
||||
from scribe.services import task_logs as task_logs_svc
|
||||
from scribe.services import trash as trash_svc
|
||||
from scribe.services.note_usage import record_pulled
|
||||
|
||||
|
||||
async def list_tasks(
|
||||
@@ -99,6 +100,14 @@ async def get_task(task_id: int) -> dict:
|
||||
data["suppressed_rules"] = applicable.get("suppressed_rules", [])
|
||||
data["suppressed_topics"] = applicable.get("suppressed_topics", [])
|
||||
data.update(await access_svc.describe_provenance(uid, note))
|
||||
# Same reasoning as get_note's record_pulled, and this is the tool where it
|
||||
# matters MOST: auto-inject ranks kind-blind over a corpus that is
|
||||
# overwhelmingly tasks and issues, so tasks dominate what it surfaces. Without
|
||||
# this the surfaced→pulled loop was open exactly where the volume is — every
|
||||
# surfaced task counted as never-pulled because the tool that opens one didn't
|
||||
# say so, driving auto-inject's measured pull-through toward zero for its own
|
||||
# dominant kind. #1038 and #2085 are explicitly gated on that number (#2245).
|
||||
record_pulled(user_id=uid, note_id=int(note.id), source="mcp_get_task")
|
||||
return data
|
||||
|
||||
|
||||
|
||||
@@ -186,3 +186,37 @@ async def test_auto_inject_records_what_survived_the_margin_gate():
|
||||
|
||||
assert surfaced.call_args.kwargs["note_ids"] == [1]
|
||||
assert surfaced.call_args.kwargs["source"] == "auto_inject"
|
||||
|
||||
|
||||
def test_every_getter_that_can_be_surfaced_also_records_a_pull():
|
||||
"""Rule #33 contract check — and the one that would have caught #2245.
|
||||
|
||||
`surfaced` and `pulled` only mean something as a PAIR: the rate between them
|
||||
is what #1038 and #2085 gate on. That pair is only closed if the tool which
|
||||
OPENS a record reports it. `get_note` did, `get_snippet` did, `get_task` did
|
||||
NOT — and auto-inject ranks kind-blind over a corpus that is overwhelmingly
|
||||
tasks and issues, so the gap sat exactly where the volume is: every surfaced
|
||||
task counted as never-pulled, dragging measured pull-through toward zero for
|
||||
the menu's own dominant kind.
|
||||
|
||||
Asserted by source inspection rather than by calling the tools, because the
|
||||
failure is a MISSING call — which no behavioural test of the tool's return
|
||||
value can see.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
from scribe.mcp.tools import notes as notes_tools
|
||||
from scribe.mcp.tools import snippets as snippet_tools
|
||||
from scribe.mcp.tools import tasks as task_tools
|
||||
|
||||
getters = (
|
||||
(notes_tools, "get_note"),
|
||||
(task_tools, "get_task"),
|
||||
(snippet_tools, "get_snippet"),
|
||||
)
|
||||
for module, name in getters:
|
||||
src = inspect.getsource(getattr(module, name))
|
||||
assert "record_pulled(" in src, (
|
||||
f"{name} can be surfaced in an auto-inject menu but records no pull — "
|
||||
"its pull-through rate will read as zero regardless of real usage"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user