feat(telemetry): record WHAT the bar turned away, not only how close it came (#3807)
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / integration (push) Successful in 40s
CI & Build / TypeScript typecheck (push) Successful in 43s
CI & Build / Python tests (push) Successful in 1m14s
CI & Build / Build & push image (push) Successful in 2m59s

#3670 added `best_available_score` so a threshold could be judged from its
rejections. It records how CLOSE the bar came to firing and not WHAT it
refused, and that is the half a decision actually needs.

Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 —
about 117 declines a day within 0.013 of firing. Dropping to 0.707 would
take that arm from 22 hits a day to roughly 139: six-fold, on a surface
that runs before every Bash call. The percentile says the mass is there.
Nothing said whether it was worth showing.

NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected
rule line already carries title and trigger, so a session can comply
without ever calling get_rule, and rule pull-through understates
usefulness by construction. Reading the rejected records can — and
`result_ids` holds only what was RETURNED, so on a zero-result call the
near-missed record had no name at all.

So the id, from the SAME ranked candidate as the score. Both searches
unpack `best` once and read both fields off it, because splitting that
into two expressions is exactly how a later edit pairs a score with its
neighbour's id — and a score attached to the wrong record is worse than no
id, since it invites judging the wrong one and concluding the bar is fine.

write_path withholds the id on the same condition it withholds the score
(#3739): a surviving id beside a null score names a record without saying
what it scored, the pair disagreeing in the other direction.

THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled,
and a reader tuning a bar needs to go and read the records. Opt-in via
`near_miss_samples` (0-20, default 0) so the ordinary readout keeps its
size, and deliberately NOT a window function: this module's one production
outage was a grouped query Postgres rejected, swallowed by the broad
except, every counter reading zero while the mocked tests passed (#2663).
One flat ordered query, overfetched, bucketed in Python — the shape that
lesson prescribes.

Migration 0097, nullable and unbackfilled. Not a foreign key: the table
spans record types and `source` says which, exactly as result_ids works.

The integration guard pins the listing as PER SOURCE. A global LIMIT would
let a noisy source eat the whole quota and leave the surface being tuned
showing nothing — which reads as "nothing was close", the misreading this
milestone has spent itself correcting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-09 21:24:32 -04:00
co-authored by Claude Opus 5
parent 623464323e
commit d5ac8408f6
8 changed files with 287 additions and 5 deletions
+10 -2
View File
@@ -658,7 +658,12 @@ async def semantic_search_notes(
# looked" and from "the query failed". Set here, at the one point past
# which a real result set exists.
report["searched"] = True
report["best_available_score"] = scored[0][0] if scored else None
# ONE unpack, so the score and the id cannot describe different records
# (#3807). Splitting these into two expressions is how a later edit
# pairs a score with its neighbour's id.
best = scored[0] if scored else None
report["best_available_score"] = best[0] if best else None
report["best_available_id"] = int(best[1].id) if best else None
scored = [pair for pair in scored if pair[0] >= threshold]
if not demote_superseded:
return scored[:limit]
@@ -909,7 +914,10 @@ async def semantic_search_rules(
if report is not None:
# See the sibling search: absent means the search never ran (#3765).
report["searched"] = True
report["best_available_score"] = ranked[0][0] if ranked else None
# One unpack — see the sibling search (#3807).
best = ranked[0] if ranked else None
report["best_available_score"] = best[0] if best else None
report["best_available_id"] = int(best[1].id) if best else None
return [pair for pair in ranked if pair[0] >= threshold][:limit]