feat(telemetry): retrieval_telemetry reports rule pull-through where it reported nothing (#3317)
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / integration (push) Successful in 32s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 29s

Milestone 333 step 3, the read half. Steps 1 and 2 built the table and filled
it; until now nothing read it, and `usage` — sourced entirely from
note_usage_events — described notes only while `sources` happily listed a
write_path_rule row above it. A reader takes the aggregate as covering
everything named above it. It did not.

A SEPARATE `rule_usage` BLOCK, not folded into `usage`. Two reasons, and the
second is the one that bites: the corpora differ by orders of magnitude, so a
blended ratio would be the note ratio with noise on it and the rule arm would
stay invisible inside it; and `usage` is what existing callers already read and
compare across windows, so silently changing what it counts would move a number
nobody was told had changed meaning. There is a test asserting rule events stay
out of the note block.

No `ambient` key, unlike the twin. Nothing surfaces a rule un-ranked —
list_always_on_rules and enter_project hand rules over wholesale but emit no
event — so there is no ambient class to subtract. The absence is a fact about
the data, not an oversight, and it returns when a bulk loader starts emitting.

Guarded separately, like `by_source`. This table did not exist a commit ago,
and an instance running upgraded code against un-migrated schema would
otherwise take down two readouts that work perfectly in order to report a third
that cannot. On failure the FLAG is added and the SHAPE is kept — a caller must
not have to choose between crashing on a missing key and quietly rendering
zeros it has no right to.

`pull_through` is None rather than 0.0 on an empty window, matching the note
block. A ratio of zero asserts "rules were shown and none opened"; with an
empty numerator and denominator that is a claim the data does not support, and
it is the reading that would make a brand-new install look like a broken one.

Also fixed, from #3311: the rule arm never timed its search, so it was the one
source in the readout reporting a null p90_duration_ms — a gap that reads as
"this surface is somehow not measurable" rather than "nobody passed the
number".

Both docstrings updated in the same change. The tool's is the agent-facing
contract (rule 119) and it explicitly said rule surfacings were absent and had
"no usage counter at all". Leaving that would have had a reader conclude the
arm has zero pull-through rather than a separate one.

Tests are integration for the reason the block above them is: real GROUP BYs
and count(distinct) against a table a commit old, in a module whose one
production outage was a SQL shape the database rejected inside a broad except.
A mock would agree with whatever the code does, including nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
2026-09-02 17:25:11 -04:00
co-authored by Claude Opus 5
parent 70761b16d9
commit 8901c904a9
4 changed files with 311 additions and 13 deletions
+30 -10
View File
@@ -158,7 +158,7 @@ async def retrieval_telemetry(days: int = 30) -> dict:
hand-probing the live instance, which is how the last such decision had to
be made.
Two readouts, from the two tables built for them:
Three readouts, from the three tables built for them:
`sources` — per retrieval surface (`auto_inject`, `write_path`,
`mcp_search`, …), from `retrieval_logs`: `calls`, `zero_result_calls`,
@@ -168,7 +168,7 @@ async def retrieval_telemetry(days: int = 30) -> dict:
against `calls`, with the spread beside it: a surface that clears its bar
on nearly every call is either well-tuned or too loose, and p10 says which.
`usage` — from `note_usage_events`, at the per-note grain
`usage` — NOTES ONLY, from `note_usage_events`, at the per-note grain
`retrieval_logs` cannot be indexed at: `surfaced` (ranked surfacings — a
scored surface CHOSE the record), `ambient` (the rest), `pulled` split into
`pulled_by_agent` / `pulled_by_human`, the distinct-note counts, and
@@ -190,14 +190,34 @@ async def retrieval_telemetry(days: int = 30) -> dict:
about a record nothing chose). Read it as: of the distinct notes THIS
surface put in front of the agent, how many did the agent then open?
Two limits on it, both deliberate. It is an UPPER BOUND per surface: a pull
records the door it came through, not the surface that led there, so a note
surfaced by two surfaces and opened once counts for both — attribution
would need the session identity #2085 declined to invent. And RULE
surfacings are absent: `write_path_rule` appears in `sources` with its
scores but has no usage counter at all, so it has no row here (#3311).
`by_source_failed: true` means that one query failed while the rest of the
readout stood.
It is an UPPER BOUND per surface: a pull records the door it came
through, not the surface that led there, so a note surfaced by two surfaces
and opened once counts for both — attribution would need the session
identity #2085 declined to invent. `by_source_failed: true` means that one
query failed while the rest of the readout stood.
`rule_usage` — the same question for RULES, from `rule_usage_events`:
`surfaced`, `pulled` split into `pulled_by_agent` / `pulled_by_human`, the
distinct-rule counts, and `pull_through` on the same definition (agent
pulls over surfacings).
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
with that is the mistake to avoid. The corpora differ by orders of
magnitude — a few dozen eligible rules against thousands of notes — so a
blended ratio would be the note ratio with noise on it and would hide the
rule arm entirely. It also has no `ambient` key, because nothing surfaces a
rule un-ranked: `list_always_on_rules` and `enter_project` hand over rules
wholesale but emit no event, so there is no ambient class to separate.
Read it against `sources["write_path_rule"]`. That surface has never once
declined to fire, and until this block existed there was no way to tell a
well-tuned arm from a bar it cannot fail to clear (#3311). `pull_through`
is the number that tells them apart.
`rule_usage_failed: true` means that read failed while the rest of the
readout stood. The counts are still present so a caller can render, but
they are zeros meaning "could not find out", not "nothing happened" — do
not report a pull-through from a block carrying that flag.
Scoped to your own telemetry — a retrieval log records what your agent
asked for, query text included, and is not a shared record kind.