feat(telemetry): retrieval_telemetry reports rule pull-through where it reported nothing (#3317)
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / integration (push) Successful in 32s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 29s

Milestone 333 step 3, the read half. Steps 1 and 2 built the table and filled
it; until now nothing read it, and `usage` — sourced entirely from
note_usage_events — described notes only while `sources` happily listed a
write_path_rule row above it. A reader takes the aggregate as covering
everything named above it. It did not.

A SEPARATE `rule_usage` BLOCK, not folded into `usage`. Two reasons, and the
second is the one that bites: the corpora differ by orders of magnitude, so a
blended ratio would be the note ratio with noise on it and the rule arm would
stay invisible inside it; and `usage` is what existing callers already read and
compare across windows, so silently changing what it counts would move a number
nobody was told had changed meaning. There is a test asserting rule events stay
out of the note block.

No `ambient` key, unlike the twin. Nothing surfaces a rule un-ranked —
list_always_on_rules and enter_project hand rules over wholesale but emit no
event — so there is no ambient class to subtract. The absence is a fact about
the data, not an oversight, and it returns when a bulk loader starts emitting.

Guarded separately, like `by_source`. This table did not exist a commit ago,
and an instance running upgraded code against un-migrated schema would
otherwise take down two readouts that work perfectly in order to report a third
that cannot. On failure the FLAG is added and the SHAPE is kept — a caller must
not have to choose between crashing on a missing key and quietly rendering
zeros it has no right to.

`pull_through` is None rather than 0.0 on an empty window, matching the note
block. A ratio of zero asserts "rules were shown and none opened"; with an
empty numerator and denominator that is a claim the data does not support, and
it is the reading that would make a brand-new install look like a broken one.

Also fixed, from #3311: the rule arm never timed its search, so it was the one
source in the readout reporting a null p90_duration_ms — a gap that reads as
"this surface is somehow not measurable" rather than "nobody passed the
number".

Both docstrings updated in the same change. The tool's is the agent-facing
contract (rule 119) and it explicitly said rule surfacings were absent and had
"no usage counter at all". Leaving that would have had a reader conclude the
arm has zero pull-through rather than a separate one.

Tests are integration for the reason the block above them is: real GROUP BYs
and count(distinct) against a table a commit old, in a module whose one
production outage was a SQL shape the database rejected inside a broad except.
A mock would agree with whatever the code does, including nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
2026-09-02 17:25:11 -04:00
co-authored by Claude Opus 5
parent 70761b16d9
commit 8901c904a9
4 changed files with 311 additions and 13 deletions
+7 -1
View File
@@ -1126,10 +1126,16 @@ async def build_write_path_hint(
rule_ids: list[int] = []
try:
already = set(exclude_rule_ids or [])
# Timed like the notes arm above. Without this the rule row was the one
# source in the whole readout reporting a null p90_duration_ms (#3311)
# — a gap that reads as "this surface is somehow not measurable" rather
# than "nobody passed the number".
rule_t0 = time.perf_counter()
hits = await semantic_search_rules(
user_id, code or path, limit=2,
threshold=cfg["threshold"], tier="conditional",
)
rule_ms = (time.perf_counter() - rule_t0) * 1000.0
fresh = [(score, rule) for score, rule in hits if rule.id not in already]
for _score, rule in fresh:
trigger = (rule.when_to_apply or "").strip()
@@ -1155,7 +1161,7 @@ async def build_write_path_hint(
record_retrieval(
user_id=user_id, source="write_path_rule", query=code or path,
threshold=cfg["threshold"], limit=2, project_id=project_id,
is_task=None, results=fresh,
is_task=None, results=fresh, duration_ms=rule_ms,
)
# `rule_ids` is `fresh`, i.e. AFTER exclude_rule_ids. A rule the
# session already holds was considered and not shown, and counting