fix(telemetry): both rule arms logged only their hits, so the clear-rate could only read 100% (#3497)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / integration (push) Successful in 33s
CI & Build / Python tests (push) Failing after 48s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / integration (push) Successful in 33s
CI & Build / Python tests (push) Failing after 48s
CI & Build / Build & push image (push) Skipped
`write_path_rule` reported `zero_result_calls: 0` and `cleared_threshold: 133/133` — a perfect record no other surface comes near (`write_path` 421 zeroes of 613, `reuse_slot` 124/199, `auto_inject` 114/326). #3311 read that as a measurement and milestone 333 was scoped on it. It was an artifact. Both arms called `record_retrieval` inside a guard on having results — the write-path arm behind `if fresh:`, the pre-tool arm below `if not fresh: return out` — so a call that found nothing wrote no row. The statistic was a fact about the shape of the code, true at any threshold whatsoever. The call log moves out of the guard in both arms. The surfacing log stays in it: nothing was shown, so no surfacing occurred. `results=fresh` is kept deliberately — the note arms pass exclusions into `semantic_search_notes`, so what they log is already post-exclusion, and logging `hits` here would make this row mean something other than every other row in the same readout. The defect bites hardest on the pre-tool arm, which fires on every Bash call: with no rows at all, a ranker that declined is indistinguishable from a hook that never fired — the silent failure the arm exists to stop. Tests cover both arms behaviourally (found nothing; found only what the session already held; searched nothing at all, which must stay silent) plus a structural guard, because this was one level of indentation and it appeared independently in two places. #3311 and the `rule_usage` docstring corrected rather than quietly rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -12,11 +12,23 @@ Two event streams, deliberately independent:
|
||||
|
||||
WHY THIS ARM AND NOT ANOTHER. Every other surface declines most of the time —
|
||||
`write_path` returns nothing on 78% of calls, `reuse_slot` on 79%, auto-inject
|
||||
on 39%. The rule arm has never once returned nothing (#3311). That is either a
|
||||
perfectly tuned surface or a bar it cannot fail to clear, and `retrieval_logs`
|
||||
cannot tell the two apart: it records what the ranker scored, never whether the
|
||||
hint was any use. The ratio these two streams produce is the missing half, and
|
||||
without it any threshold change is a number picked off a histogram.
|
||||
on 39%. The rule arm APPEARED never to have returned nothing (#3311), and this
|
||||
docstring used to put that forward as the puzzle worth measuring: "either a
|
||||
perfectly tuned surface or a bar it cannot fail to clear".
|
||||
|
||||
It was neither, and the correction belongs here rather than being quietly
|
||||
deleted. The arm wrote its `retrieval_logs` row only on calls that FOUND
|
||||
something (#3497), so `zero_result_calls` sat at 0 and `cleared_threshold` at
|
||||
`calls` because of the shape of the code — at any threshold whatsoever. A
|
||||
statistic that could not vary was read as a finding about the corpus. It is the
|
||||
#2663 failure mode one level up: there the broken readout was a zero, here it
|
||||
was a hundred percent, which is far better camouflage.
|
||||
|
||||
The reason to measure this arm survives the correction, and is stronger for it.
|
||||
`retrieval_logs` records what the ranker scored, never whether the hint was any
|
||||
use, so even an honest clear-rate would not settle the question. The ratio these
|
||||
two streams produce is the missing half, and without it any threshold change is
|
||||
a number picked off a histogram.
|
||||
|
||||
Design notes, mirroring `note_usage`:
|
||||
- Writes are fire-and-forget through `background.spawn`, so telemetry never
|
||||
|
||||
Reference in New Issue
Block a user