feat(telemetry): the rule arm records what it showed, and get_rule records the read (#3316)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 24s
CI & Build / integration (push) Successful in 37s
CI & Build / Python tests (push) Failing after 53s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 24s
CI & Build / integration (push) Successful in 37s
CI & Build / Python tests (push) Failing after 53s
CI & Build / Build & push image (push) Skipped
Milestone 333 step 2. Step 1 built the table; a counter nobody calls reads zero and looks exactly like a surface nobody uses, which is #2663's shape. SURFACED — the standing-rule arm in build_write_path_hint, beside the record_retrieval it already made. Two tables, and the split is not arbitrary: retrieval_logs is one row per CALL keyed on the score distribution a threshold is tuned from; rule_usage_events is one row per RULE per event, the grain "was this hint ever acted on" needs and the grain a JSONB result_ids array cannot be indexed at. The comment there said rule ids had nowhere to go — that note_usage_events remaps ids on restore, so a rule id would return attached to whatever note took that number. Still true of the NOTE table, and precisely why step 1 built its own. Rewritten to say the gap is closed rather than leaving a stale rationale that would have someone re-derive the same dead end. Records `fresh`, i.e. AFTER exclude_rule_ids. A rule the session already holds was considered and not shown; counting it would inflate the denominator with claims the agent never saw, and the ratio would then fall for a reason that has nothing to do with whether hints land. PULLED — two doors, both after their access check so a refused read is not a pull. mcp_get_rule is the one that matters: the arm's own message ends "Read it with get_rule(N)", so that call is the exact action a landed hint produces. rest_rule carries the other prefix, and the prefix is load-bearing — "is this rule dead weight?" is served by any pull, "did that injected hint land?" by agent pulls only. NOT a pull: rule_history. It loads the rule for its title and its own output says "The current wording is on the rule itself — get_rule(N)", so counting it would credit a read of the history as a read of the rule and double-count anyone who then follows that pointer. list_always_on_rules and enter_project are likewise bulk resident loads, not somebody choosing to open one record. tests/test_rule_usage_wiring.py is cross-cutting on purpose: the surfaced end is in plugin_context, the pull end in two other modules, and "both ends meet" is a property no module-shaped file asserts. It covers the exclusion boundary, that a failing recorder cannot break the write, that a refused read records nothing, and two completeness guards — every door records, and the bulk loaders still do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
@@ -32,6 +32,7 @@ from scribe.services import snippets as snippets_svc
|
||||
from scribe.services.access import label_shared_items, owner_names_for
|
||||
from scribe.services.embeddings import semantic_search_notes, semantic_search_rules
|
||||
from scribe.services.note_usage import record_surfaced
|
||||
from scribe.services.rule_usage import record_rule_surfaced
|
||||
from scribe.services.supersession import superseded_ids
|
||||
from scribe.services.retrieval_telemetry import record_retrieval
|
||||
from scribe.services.settings import get_setting
|
||||
@@ -1140,15 +1141,29 @@ async def build_write_path_hint(
|
||||
)
|
||||
rule_ids.append(rule.id)
|
||||
if fresh:
|
||||
# retrieval_logs, NOT note_usage_events: that table's ids are
|
||||
# remapped on a backup restore, so a rule id there would return
|
||||
# attached to whatever note took that number. This one is never
|
||||
# restored, and `source` already separates the surfaces.
|
||||
# TWO tables, and the split is not arbitrary. retrieval_logs is one
|
||||
# row per CALL, keyed on the score distribution a threshold is
|
||||
# tuned from. rule_usage_events is one row per RULE per event,
|
||||
# which is the grain "was this hint ever acted on" needs and the
|
||||
# grain a JSONB result_ids array cannot be indexed at.
|
||||
#
|
||||
# This comment used to say rule ids had nowhere to go — that
|
||||
# note_usage_events remaps ids on restore, so a rule id there would
|
||||
# return attached to whatever note took that number. That is still
|
||||
# true of the NOTE table, and it is exactly why rule_usage_events
|
||||
# is its own (milestone 333 step 1). The gap it described is closed.
|
||||
record_retrieval(
|
||||
user_id=user_id, source="write_path_rule", query=code or path,
|
||||
threshold=cfg["threshold"], limit=2, project_id=project_id,
|
||||
is_task=None, results=fresh,
|
||||
)
|
||||
# `rule_ids` is `fresh`, i.e. AFTER exclude_rule_ids. A rule the
|
||||
# session already holds was considered and not shown, and counting
|
||||
# it would inflate the denominator with claims the agent never saw
|
||||
# — which reads as a precision problem this arm does not have.
|
||||
record_rule_surfaced(
|
||||
user_id=user_id, rule_ids=rule_ids, source="write_path_rule",
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("write-path rule arm failed", exc_info=True)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user