fix(telemetry): a repeat is not a rejection, and near_misses counted it as one (#3739)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / integration (push) Successful in 45s
CI & Build / Python tests (push) Successful in 1m27s
CI & Build / Build & push image (push) Successful in 31s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / integration (push) Successful in 45s
CI & Build / Python tests (push) Successful in 1m27s
CI & Build / Build & push image (push) Successful in 31s
Caught on the first live read after deploying #3670. The readout contradicted itself: pre_tool_rule top_score.min 0.7204 the lowest score ever RETURNED near_misses.max 0.7457 "rejected", but scored higher `best_available_score` is measured pre-threshold, which is right, but for the rule arms it is also PRE-EXCLUSION, which is not. The note arms pass `exclude_ids` into semantic_search_notes so their score is already post-exclusion and clean; `semantic_search_rules` takes no such parameter, so the rule arms filter in Python after the search and a rule that cleared the bar and was dropped as a repeat still reported its score on a zero-result row. That is #3497's distinction — a ranker decline versus a reader already ahead of it — reintroduced one level up, inside the field built to replace a tautology. The population now also requires `suppressed_count IS NULL OR = 0`. The NULL arm is principled rather than permissive: null means the caller filtered INSIDE the search, which is exactly the case where the reported score cannot be contaminated. Deliberately conservative — a call carrying both a repeat and a lower genuine miss is dropped whole, losing that point. It undercounts; it cannot corrupt, which is the right way round for a number read against a bar. It also makes `near_misses.max < threshold` true BY CONSTRUCTION rather than by fixture: an above-bar candidate nobody excluded would have been returned, so its call is not in the population at all. THE TEST DID NOT CATCH THIS, and that is the part worth keeping. The assertion `nm["max"] < 0.72` was already there, with exactly the right intent. It passed because the fixture contained no suppressed call — the guard held because the breaking shape was absent, not because the code was right. Rule 167's stated failure mode, in a test written while citing rule 167. The fixture now builds that shape: a 0.9 hit dropped as a repeat, which lands in the population and drags `max` above the threshold unless the predicate excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -169,10 +169,12 @@ async def retrieval_telemetry(days: int = 30) -> dict:
|
||||
`avg_result_count` and `p90_duration_ms`.
|
||||
|
||||
THE NUMBER TO READ FIRST IS `near_misses.p90`, AGAINST THE THRESHOLD IN
|
||||
FORCE FOR THAT SURFACE. It is measured only on the calls that returned
|
||||
NOTHING, on the best score the ranker reached before the bar rejected it —
|
||||
so it is the one figure here that says something the bar cannot make true
|
||||
by construction. A bar at 0.72 turning away a stream of 0.71s is set too
|
||||
FORCE FOR THAT SURFACE. It is measured on the calls the BAR turned away —
|
||||
zero-result calls, minus the ones whose zero was a repeat the reader had
|
||||
already been shown — using the best score the ranker reached before the bar
|
||||
rejected it. So it is the one figure here that says something the bar
|
||||
cannot make true by construction, and `max` is always below the threshold:
|
||||
an above-bar candidate nobody excluded would have been returned. A bar at 0.72 turning away a stream of 0.71s is set too
|
||||
high by a hair and the surface is losing hits it should have had. The same
|
||||
bar turning away 0.30s is working, and the corpus simply had nothing. Both
|
||||
render as a zero-result call, and nothing else in this readout tells them
|
||||
|
||||
Reference in New Issue
Block a user