feat(telemetry): retrieval_telemetry reports rule pull-through where it reported nothing (#3317)
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / integration (push) Successful in 32s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 29s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / integration (push) Successful in 32s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 29s
Milestone 333 step 3, the read half. Steps 1 and 2 built the table and filled it; until now nothing read it, and `usage` — sourced entirely from note_usage_events — described notes only while `sources` happily listed a write_path_rule row above it. A reader takes the aggregate as covering everything named above it. It did not. A SEPARATE `rule_usage` BLOCK, not folded into `usage`. Two reasons, and the second is the one that bites: the corpora differ by orders of magnitude, so a blended ratio would be the note ratio with noise on it and the rule arm would stay invisible inside it; and `usage` is what existing callers already read and compare across windows, so silently changing what it counts would move a number nobody was told had changed meaning. There is a test asserting rule events stay out of the note block. No `ambient` key, unlike the twin. Nothing surfaces a rule un-ranked — list_always_on_rules and enter_project hand rules over wholesale but emit no event — so there is no ambient class to subtract. The absence is a fact about the data, not an oversight, and it returns when a bulk loader starts emitting. Guarded separately, like `by_source`. This table did not exist a commit ago, and an instance running upgraded code against un-migrated schema would otherwise take down two readouts that work perfectly in order to report a third that cannot. On failure the FLAG is added and the SHAPE is kept — a caller must not have to choose between crashing on a missing key and quietly rendering zeros it has no right to. `pull_through` is None rather than 0.0 on an empty window, matching the note block. A ratio of zero asserts "rules were shown and none opened"; with an empty numerator and denominator that is a claim the data does not support, and it is the reading that would make a brand-new install look like a broken one. Also fixed, from #3311: the rule arm never timed its search, so it was the one source in the readout reporting a null p90_duration_ms — a gap that reads as "this surface is somehow not measurable" rather than "nobody passed the number". Both docstrings updated in the same change. The tool's is the agent-facing contract (rule 119) and it explicitly said rule surfacings were absent and had "no usage counter at all". Leaving that would have had a reader conclude the arm has zero pull-through rather than a separate one. Tests are integration for the reason the block above them is: real GROUP BYs and count(distinct) against a table a commit old, in a module whose one production outage was a SQL shape the database rejected inside a broad except. A mock would agree with whatever the code does, including nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
@@ -158,7 +158,7 @@ async def retrieval_telemetry(days: int = 30) -> dict:
|
||||
hand-probing the live instance, which is how the last such decision had to
|
||||
be made.
|
||||
|
||||
Two readouts, from the two tables built for them:
|
||||
Three readouts, from the three tables built for them:
|
||||
|
||||
`sources` — per retrieval surface (`auto_inject`, `write_path`,
|
||||
`mcp_search`, …), from `retrieval_logs`: `calls`, `zero_result_calls`,
|
||||
@@ -168,7 +168,7 @@ async def retrieval_telemetry(days: int = 30) -> dict:
|
||||
against `calls`, with the spread beside it: a surface that clears its bar
|
||||
on nearly every call is either well-tuned or too loose, and p10 says which.
|
||||
|
||||
`usage` — from `note_usage_events`, at the per-note grain
|
||||
`usage` — NOTES ONLY, from `note_usage_events`, at the per-note grain
|
||||
`retrieval_logs` cannot be indexed at: `surfaced` (ranked surfacings — a
|
||||
scored surface CHOSE the record), `ambient` (the rest), `pulled` split into
|
||||
`pulled_by_agent` / `pulled_by_human`, the distinct-note counts, and
|
||||
@@ -190,14 +190,34 @@ async def retrieval_telemetry(days: int = 30) -> dict:
|
||||
about a record nothing chose). Read it as: of the distinct notes THIS
|
||||
surface put in front of the agent, how many did the agent then open?
|
||||
|
||||
Two limits on it, both deliberate. It is an UPPER BOUND per surface: a pull
|
||||
records the door it came through, not the surface that led there, so a note
|
||||
surfaced by two surfaces and opened once counts for both — attribution
|
||||
would need the session identity #2085 declined to invent. And RULE
|
||||
surfacings are absent: `write_path_rule` appears in `sources` with its
|
||||
scores but has no usage counter at all, so it has no row here (#3311).
|
||||
`by_source_failed: true` means that one query failed while the rest of the
|
||||
readout stood.
|
||||
It is an UPPER BOUND per surface: a pull records the door it came
|
||||
through, not the surface that led there, so a note surfaced by two surfaces
|
||||
and opened once counts for both — attribution would need the session
|
||||
identity #2085 declined to invent. `by_source_failed: true` means that one
|
||||
query failed while the rest of the readout stood.
|
||||
|
||||
`rule_usage` — the same question for RULES, from `rule_usage_events`:
|
||||
`surfaced`, `pulled` split into `pulled_by_agent` / `pulled_by_human`, the
|
||||
distinct-rule counts, and `pull_through` on the same definition (agent
|
||||
pulls over surfacings).
|
||||
|
||||
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
|
||||
with that is the mistake to avoid. The corpora differ by orders of
|
||||
magnitude — a few dozen eligible rules against thousands of notes — so a
|
||||
blended ratio would be the note ratio with noise on it and would hide the
|
||||
rule arm entirely. It also has no `ambient` key, because nothing surfaces a
|
||||
rule un-ranked: `list_always_on_rules` and `enter_project` hand over rules
|
||||
wholesale but emit no event, so there is no ambient class to separate.
|
||||
|
||||
Read it against `sources["write_path_rule"]`. That surface has never once
|
||||
declined to fire, and until this block existed there was no way to tell a
|
||||
well-tuned arm from a bar it cannot fail to clear (#3311). `pull_through`
|
||||
is the number that tells them apart.
|
||||
|
||||
`rule_usage_failed: true` means that read failed while the rest of the
|
||||
readout stood. The counts are still present so a caller can render, but
|
||||
they are zeros meaning "could not find out", not "nothing happened" — do
|
||||
not report a pull-through from a block carrying that flag.
|
||||
|
||||
Scoped to your own telemetry — a retrieval log records what your agent
|
||||
asked for, query text included, and is not a shared record kind.
|
||||
|
||||
Reference in New Issue
Block a user