feat(telemetry): record WHAT the bar turned away, not only how close it came (#3807)

#3670 added `best_available_score` so a threshold could be judged from its
rejections. It records how CLOSE the bar came to firing and not WHAT it
refused, and that is the half a decision actually needs.

Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 —
about 117 declines a day within 0.013 of firing. Dropping to 0.707 would
take that arm from 22 hits a day to roughly 139: six-fold, on a surface
that runs before every Bash call. The percentile says the mass is there.
Nothing said whether it was worth showing.

NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected
rule line already carries title and trigger, so a session can comply
without ever calling get_rule, and rule pull-through understates
usefulness by construction. Reading the rejected records can — and
`result_ids` holds only what was RETURNED, so on a zero-result call the
near-missed record had no name at all.

So the id, from the SAME ranked candidate as the score. Both searches
unpack `best` once and read both fields off it, because splitting that
into two expressions is exactly how a later edit pairs a score with its
neighbour's id — and a score attached to the wrong record is worse than no
id, since it invites judging the wrong one and concluding the bar is fine.

write_path withholds the id on the same condition it withholds the score
(#3739): a surviving id beside a null score names a record without saying
what it scored, the pair disagreeing in the other direction.

THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled,
and a reader tuning a bar needs to go and read the records. Opt-in via
`near_miss_samples` (0-20, default 0) so the ordinary readout keeps its
size, and deliberately NOT a window function: this module's one production
outage was a grouped query Postgres rejected, swallowed by the broad
except, every counter reading zero while the mocked tests passed (#2663).
One flat ordered query, overfetched, bucketed in Python — the shape that
lesson prescribes.

Migration 0097, nullable and unbackfilled. Not a foreign key: the table
spans record types and `source` says which, exactly as result_ids works.

The integration guard pins the listing as PER SOURCE. A global LIMIT would
let a noisy source eat the whole quota and leave the surface being tuned
showing nothing — which reads as "nothing was close", the misreading this
milestone has spent itself correcting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/mcp/tools/search.py

@@ -128,6 +128,7 @@ async def search(
         project_id=project_id or None, is_task=is_task, results=raw,
         duration_ms=(time.perf_counter() - t0) * 1000.0,
         best_available=report.get("best_available_score"),
+        best_available_id=report.get("best_available_id"),
         searched=bool(report.get("searched", True)),
     )
     owners = await owner_names_for(
@@ -153,7 +154,9 @@ async def search(
     }
 
 
-async def retrieval_telemetry(days: int = 30) -> dict:
+async def retrieval_telemetry(
+    days: int = 30, near_miss_samples: int = 0,
+) -> dict:
     """What the retrieval telemetry says about YOUR surfaces, over a window.
 
     The read half of the loop the ranker's thresholds are meant to be tuned
@@ -181,6 +184,24 @@ async def retrieval_telemetry(days: int = 30) -> dict:
     render as a zero-result call, and nothing else in this readout tells them
     apart.
 
+    `near_miss_samples` (0-20, default 0) TURNS THE PERCENTILES INTO RECORDS
+    YOU CAN READ. Each source then carries `near_miss_records`: its highest
+    scoring declines, each with the `record_id` the bar refused and the `query`
+    that asked. Reach for it whenever you are about to move a threshold.
+
+    THE PERCENTILE CANNOT SETTLE A BAR ON ITS OWN, and this is the whole reason
+    the parameter exists. `near_misses.p90` says mass is sitting just under the
+    line; it says nothing about whether that mass is RELEVANT, and those are
+    different questions. Lowering a bar to where the mass is, without reading
+    what is there, is choosing a firing rate rather than a quality. Pull-through
+    cannot referee it either — the injected rule line already carries title and
+    trigger, so a session can comply without ever calling `get_rule`, which
+    makes rule pull-through understate usefulness by construction. Reading the
+    rejected records is the method that actually answers it.
+
+    Off by default because it is a LISTING, not a statistic: it is for the
+    moment you are making a decision, not for every readout.
+
     `near_misses` is `null` when no declining call in the window measured it —
     rows written before #3670 shipped cannot know. That is "not measured", not
     "nothing came close"; a 0.0 there would be a claim about the corpus
@@ -320,8 +341,15 @@ It is an UPPER BOUND per surface: a pull records the door it came
 
     Args:
         days: window size, default 30. Clamped to at least 1.
+        near_miss_samples: 0-20, default 0. How many of each source's highest
+            scoring DECLINES to list by record, with the query that asked.
+            Pass it when you are about to move a threshold; leave it off
+            otherwise. See the near-miss section above for why a percentile
+            alone cannot settle a bar.
     """
-    return await retrieval_summary(current_user_id(), days=days)
+    return await retrieval_summary(
+        current_user_id(), days=days, near_miss_samples=near_miss_samples,
+    )
 
 
 def register(mcp) -> None: