One commit: d5ac840. CI run 6220 green on the head SHA.
0096 made a threshold judgeable from what it rejected — but it records how CLOSE the bar came to firing and not WHAT it turned away, and that is the half a decision actually needs.
Live, pre_tool_rule runs a bar of ~0.72 against a near-miss p90 of 0.7071 — roughly 117 declines a day sitting within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to about 139, a six-fold change on a surface that runs before every Bash call. The percentile says the mass is there. Nothing says whether it is worth showing.
Neither obvious instrument answers it. Pull-through cannot referee a rule: the injected line already carries title and trigger, so a session complies without ever calling get_rule, and rule pull-through understates usefulness by construction. Reading the rejected records can — and result_ids holds only what was RETURNED, so on a zero-result call it is empty and the near-missed record has no name.
What lands
Migration 0097 — retrieval_logs.best_available_id, Integer, nullable, unbackfilled. Deliberately not a foreign key: the table spans record types (rule arms store rule ids, note arms store note ids) and source is what says which table an id belongs to, exactly as result_ids has always worked. A constraint would have to point at one table and be wrong for the other.
Both searches unpack the top candidate once — best = ranked[0], then score and id read off that single object. Two separate expressions is how a later edit pairs a score with its neighbour's id, and a score attached to the wrong record is worse than no id at all: it invites reading the wrong rule and concluding the bar is fine.
write_path withholds the id on exactly the condition it withholds the score, or the readout names a record with no score beside it — the same pair disagreeing from the other direction.
near_miss_samples (0–20, default 0) on retrieval_summary() and the retrieval_telemetry MCP tool. Opt-in, so the default payload does not grow.
The one implementation choice worth flagging
The obvious SQL for "top N per source" is row_number() OVER (PARTITION BY ...). This uses a flat ordered query with bounded overfetch, bucketed per source in Python instead.
This module's one production outage (#2663) was a grouped query Postgres rejected, swallowed by the broad except, every counter reading zero while the mocked tests passed. The lesson recorded there — group on a raw column, classify in Python — applies here more than anywhere: a window function would be the same bet again, on the same table, in the same function.
The integration guard pins the listing as per source. A global LIMIT would let a noisy source eat the quota and leave the surface being tuned showing an empty list, which reads as "nothing was close" — the exact misreading this milestone exists to prevent. Falsified against a global-limit implementation before committing.
Deploying
Migration 0097 must run. The listing returns nothing until it does. Nullable and unbackfilled, so null means "not measured", never "nothing was close".
This does not move any threshold. It builds the instrument and stops. Once a window has accumulated, retrieval_telemetry(near_miss_samples=10) names the actual rules pre_tool_rule is refusing at 0.71, and the six-fold decision becomes evidential.
One commit: `d5ac840`. CI run 6220 green on the head SHA.
0096 made a threshold judgeable from what it *rejected* — but it records how CLOSE the bar came to firing and not WHAT it turned away, and that is the half a decision actually needs.
Live, `pre_tool_rule` runs a bar of ~0.72 against a near-miss p90 of 0.7071 — roughly 117 declines a day sitting within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to about 139, a six-fold change on a surface that runs before every Bash call. The percentile says the mass is there. **Nothing says whether it is worth showing.**
Neither obvious instrument answers it. Pull-through cannot referee a rule: the injected line already carries title and trigger, so a session complies without ever calling `get_rule`, and rule pull-through understates usefulness by construction. Reading the rejected records can — and `result_ids` holds only what was RETURNED, so on a zero-result call it is empty and the near-missed record has no name.
## What lands
**Migration 0097** — `retrieval_logs.best_available_id`, Integer, nullable, unbackfilled. Deliberately **not a foreign key**: the table spans record types (rule arms store rule ids, note arms store note ids) and `source` is what says which table an id belongs to, exactly as `result_ids` has always worked. A constraint would have to point at one table and be wrong for the other.
**Both searches unpack the top candidate once** — `best = ranked[0]`, then score and id read off that single object. Two separate expressions is how a later edit pairs a score with its neighbour's id, and a score attached to the wrong record is worse than no id at all: it invites reading the wrong rule and concluding the bar is fine.
**`write_path` withholds the id on exactly the condition it withholds the score**, or the readout names a record with no score beside it — the same pair disagreeing from the other direction.
**`near_miss_samples` (0–20, default 0)** on `retrieval_summary()` and the `retrieval_telemetry` MCP tool. Opt-in, so the default payload does not grow.
## The one implementation choice worth flagging
The obvious SQL for "top N per source" is `row_number() OVER (PARTITION BY ...)`. This uses a flat ordered query with bounded overfetch, bucketed per source in Python instead.
This module's one production outage (#2663) was a grouped query Postgres rejected, swallowed by the broad `except`, every counter reading zero while the mocked tests passed. The lesson recorded there — group on a raw column, classify in Python — applies here more than anywhere: a window function would be the same bet again, on the same table, in the same function.
The integration guard pins the listing as **per source**. A global `LIMIT` would let a noisy source eat the quota and leave the surface being tuned showing an empty list, which reads as "nothing was close" — the exact misreading this milestone exists to prevent. Falsified against a global-limit implementation before committing.
## Deploying
**Migration 0097 must run.** The listing returns nothing until it does. Nullable and unbackfilled, so null means "not measured", never "nothing was close".
**This does not move any threshold.** It builds the instrument and stops. Once a window has accumulated, `retrieval_telemetry(near_miss_samples=10)` names the actual rules `pre_tool_rule` is refusing at 0.71, and the six-fold decision becomes evidential.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
#3670 added `best_available_score` so a threshold could be judged from its
rejections. It records how CLOSE the bar came to firing and not WHAT it
refused, and that is the half a decision actually needs.
Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 —
about 117 declines a day within 0.013 of firing. Dropping to 0.707 would
take that arm from 22 hits a day to roughly 139: six-fold, on a surface
that runs before every Bash call. The percentile says the mass is there.
Nothing said whether it was worth showing.
NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected
rule line already carries title and trigger, so a session can comply
without ever calling get_rule, and rule pull-through understates
usefulness by construction. Reading the rejected records can — and
`result_ids` holds only what was RETURNED, so on a zero-result call the
near-missed record had no name at all.
So the id, from the SAME ranked candidate as the score. Both searches
unpack `best` once and read both fields off it, because splitting that
into two expressions is exactly how a later edit pairs a score with its
neighbour's id — and a score attached to the wrong record is worse than no
id, since it invites judging the wrong one and concluding the bar is fine.
write_path withholds the id on the same condition it withholds the score
(#3739): a surviving id beside a null score names a record without saying
what it scored, the pair disagreeing in the other direction.
THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled,
and a reader tuning a bar needs to go and read the records. Opt-in via
`near_miss_samples` (0-20, default 0) so the ordinary readout keeps its
size, and deliberately NOT a window function: this module's one production
outage was a grouped query Postgres rejected, swallowed by the broad
except, every counter reading zero while the mocked tests passed (#2663).
One flat ordered query, overfetched, bucketed in Python — the shape that
lesson prescribes.
Migration 0097, nullable and unbackfilled. Not a foreign key: the table
spans record types and `source` says which, exactly as result_ids works.
The integration guard pins the listing as PER SOURCE. A global LIMIT would
let a noisy source eat the whole quota and leave the surface being tuned
showing nothing — which reads as "nothing was close", the misreading this
milestone has spent itself correcting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
bvandeusen
changed title from Retrieval telemetry: a readout that can no longer look correct while being wrong to Record WHAT the bar turned away, not only how close it came (#3807)2026-09-09 21:34:47 -04:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
One commit:
d5ac840. CI run 6220 green on the head SHA.0096 made a threshold judgeable from what it rejected — but it records how CLOSE the bar came to firing and not WHAT it turned away, and that is the half a decision actually needs.
Live,
pre_tool_ruleruns a bar of ~0.72 against a near-miss p90 of 0.7071 — roughly 117 declines a day sitting within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to about 139, a six-fold change on a surface that runs before every Bash call. The percentile says the mass is there. Nothing says whether it is worth showing.Neither obvious instrument answers it. Pull-through cannot referee a rule: the injected line already carries title and trigger, so a session complies without ever calling
get_rule, and rule pull-through understates usefulness by construction. Reading the rejected records can — andresult_idsholds only what was RETURNED, so on a zero-result call it is empty and the near-missed record has no name.What lands
Migration 0097 —
retrieval_logs.best_available_id, Integer, nullable, unbackfilled. Deliberately not a foreign key: the table spans record types (rule arms store rule ids, note arms store note ids) andsourceis what says which table an id belongs to, exactly asresult_idshas always worked. A constraint would have to point at one table and be wrong for the other.Both searches unpack the top candidate once —
best = ranked[0], then score and id read off that single object. Two separate expressions is how a later edit pairs a score with its neighbour's id, and a score attached to the wrong record is worse than no id at all: it invites reading the wrong rule and concluding the bar is fine.write_pathwithholds the id on exactly the condition it withholds the score, or the readout names a record with no score beside it — the same pair disagreeing from the other direction.near_miss_samples(0–20, default 0) onretrieval_summary()and theretrieval_telemetryMCP tool. Opt-in, so the default payload does not grow.The one implementation choice worth flagging
The obvious SQL for "top N per source" is
row_number() OVER (PARTITION BY ...). This uses a flat ordered query with bounded overfetch, bucketed per source in Python instead.This module's one production outage (#2663) was a grouped query Postgres rejected, swallowed by the broad
except, every counter reading zero while the mocked tests passed. The lesson recorded there — group on a raw column, classify in Python — applies here more than anywhere: a window function would be the same bet again, on the same table, in the same function.The integration guard pins the listing as per source. A global
LIMITwould let a noisy source eat the quota and leave the surface being tuned showing an empty list, which reads as "nothing was close" — the exact misreading this milestone exists to prevent. Falsified against a global-limit implementation before committing.Deploying
Migration 0097 must run. The listing returns nothing until it does. Nullable and unbackfilled, so null means "not measured", never "nothing was close".
This does not move any threshold. It builds the instrument and stops. Once a window has accumulated,
retrieval_telemetry(near_miss_samples=10)names the actual rulespre_tool_ruleis refusing at 0.71, and the six-fold decision becomes evidential.🤖 Generated with Claude Code
https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
Retrieval telemetry: a readout that can no longer look correct while being wrongto Record WHAT the bar turned away, not only how close it came (#3807)