feat(telemetry): the rule arm records what it showed, and get_rule records the read (#3316)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 24s
CI & Build / integration (push) Successful in 37s
CI & Build / Python tests (push) Failing after 53s
CI & Build / Build & push image (push) Skipped

Milestone 333 step 2. Step 1 built the table; a counter nobody calls reads zero
and looks exactly like a surface nobody uses, which is #2663's shape.

SURFACED — the standing-rule arm in build_write_path_hint, beside the
record_retrieval it already made. Two tables, and the split is not arbitrary:
retrieval_logs is one row per CALL keyed on the score distribution a threshold
is tuned from; rule_usage_events is one row per RULE per event, the grain "was
this hint ever acted on" needs and the grain a JSONB result_ids array cannot be
indexed at.

The comment there said rule ids had nowhere to go — that note_usage_events
remaps ids on restore, so a rule id would return attached to whatever note took
that number. Still true of the NOTE table, and precisely why step 1 built its
own. Rewritten to say the gap is closed rather than leaving a stale rationale
that would have someone re-derive the same dead end.

Records `fresh`, i.e. AFTER exclude_rule_ids. A rule the session already holds
was considered and not shown; counting it would inflate the denominator with
claims the agent never saw, and the ratio would then fall for a reason that has
nothing to do with whether hints land.

PULLED — two doors, both after their access check so a refused read is not a
pull. mcp_get_rule is the one that matters: the arm's own message ends "Read it
with get_rule(N)", so that call is the exact action a landed hint produces.
rest_rule carries the other prefix, and the prefix is load-bearing — "is this
rule dead weight?" is served by any pull, "did that injected hint land?" by
agent pulls only.

NOT a pull: rule_history. It loads the rule for its title and its own output
says "The current wording is on the rule itself — get_rule(N)", so counting it
would credit a read of the history as a read of the rule and double-count
anyone who then follows that pointer. list_always_on_rules and enter_project
are likewise bulk resident loads, not somebody choosing to open one record.

tests/test_rule_usage_wiring.py is cross-cutting on purpose: the surfaced end
is in plugin_context, the pull end in two other modules, and "both ends meet"
is a property no module-shaped file asserts. It covers the exclusion boundary,
that a failing recorder cannot break the write, that a refused read records
nothing, and two completeness guards — every door records, and the bulk loaders
still do not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
2026-09-02 17:15:03 -04:00
co-authored by Claude Opus 5
parent 111eef7e30
commit 8f7f447fda
4 changed files with 235 additions and 4 deletions
+19 -4
View File
@@ -32,6 +32,7 @@ from scribe.services import snippets as snippets_svc
from scribe.services.access import label_shared_items, owner_names_for
from scribe.services.embeddings import semantic_search_notes, semantic_search_rules
from scribe.services.note_usage import record_surfaced
from scribe.services.rule_usage import record_rule_surfaced
from scribe.services.supersession import superseded_ids
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.settings import get_setting
@@ -1140,15 +1141,29 @@ async def build_write_path_hint(
)
rule_ids.append(rule.id)
if fresh:
# retrieval_logs, NOT note_usage_events: that table's ids are
# remapped on a backup restore, so a rule id there would return
# attached to whatever note took that number. This one is never
# restored, and `source` already separates the surfaces.
# TWO tables, and the split is not arbitrary. retrieval_logs is one
# row per CALL, keyed on the score distribution a threshold is
# tuned from. rule_usage_events is one row per RULE per event,
# which is the grain "was this hint ever acted on" needs and the
# grain a JSONB result_ids array cannot be indexed at.
#
# This comment used to say rule ids had nowhere to go — that
# note_usage_events remaps ids on restore, so a rule id there would
# return attached to whatever note took that number. That is still
# true of the NOTE table, and it is exactly why rule_usage_events
# is its own (milestone 333 step 1). The gap it described is closed.
record_retrieval(
user_id=user_id, source="write_path_rule", query=code or path,
threshold=cfg["threshold"], limit=2, project_id=project_id,
is_task=None, results=fresh,
)
# `rule_ids` is `fresh`, i.e. AFTER exclude_rule_ids. A rule the
# session already holds was considered and not shown, and counting
# it would inflate the denominator with claims the agent never saw
# — which reads as a precision problem this arm does not have.
record_rule_surfaced(
user_id=user_id, rule_ids=rule_ids, source="write_path_rule",
)
except Exception:
logger.debug("write-path rule arm failed", exc_info=True)