fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/mcp/tools/search.py

@@ -112,6 +112,7 @@ async def search(
         return await _search_rules(uid, q, limit)
     is_task = {"note": False, "task": True}.get(content_type)  # None => any
     t0 = time.perf_counter()
+    report: dict = {}
     raw = await semantic_search_notes(
         uid, q, limit=limit, is_task=is_task,
         project_id=project_id or None,
@@ -119,12 +120,14 @@ async def search(
         # An explicit search reaches everything the operator may read, including
         # records shared with them one-to-one.
         scope="read",
+        report=report,
     )
     record_retrieval(
         user_id=uid, source="mcp_search", query=q,
         threshold=DEFAULT_SIMILARITY_THRESHOLD, limit=limit,
         project_id=project_id or None, is_task=is_task, results=raw,
         duration_ms=(time.perf_counter() - t0) * 1000.0,
+        best_available=report.get("best_available_score"),
     )
     owners = await owner_names_for(
         {int(note.user_id) for _s, note in raw if note.user_id != uid}
@@ -162,18 +165,39 @@ async def retrieval_telemetry(days: int = 30) -> dict:
 
     `sources` — per retrieval surface (`auto_inject`, `write_path`,
     `mcp_search`, …), from `retrieval_logs`: `calls`, `zero_result_calls`,
-    `cleared_threshold` (how often the best hit beat the threshold in force for
-    that call), the `top_score` spread (p10/p50/p90/min/max), `avg_result_count`
-    and `p90_duration_ms`. THE number to read first is `cleared_threshold`
-    against `calls`, with the spread beside it: a surface that clears its bar
-    on nearly every call is either well-tuned or too loose, and p10 says which.
+    `near_misses`, the `top_score` spread (p10/p50/p90/min/max),
+    `avg_result_count` and `p90_duration_ms`.
 
-    READ `cleared_threshold` AND `zero_result_calls` TOGETHER, and check
-    `suppression` before concluding anything from either. A zero-result call is
-    two different events wearing one number: the ranker found nothing above the
-    bar, or it found only what this session had already been shown. Just the
-    first is evidence the bar is too high. `suppression` splits them where the
-    surface can tell — `zero_because_already_shown` comes off
+    THE NUMBER TO READ FIRST IS `near_misses.p90`, AGAINST THE THRESHOLD IN
+    FORCE FOR THAT SURFACE. It is measured only on the calls that returned
+    NOTHING, on the best score the ranker reached before the bar rejected it —
+    so it is the one figure here that says something the bar cannot make true
+    by construction. A bar at 0.72 turning away a stream of 0.71s is set too
+    high by a hair and the surface is losing hits it should have had. The same
+    bar turning away 0.30s is working, and the corpus simply had nothing. Both
+    render as a zero-result call, and nothing else in this readout tells them
+    apart.
+
+    `near_misses` is `null` when no declining call in the window measured it —
+    rows written before #3670 shipped cannot know. That is "not measured", not
+    "nothing came close"; a 0.0 there would be a claim about the corpus
+    invented out of a caller's silence.
+
+    THERE IS NO `cleared_threshold` ANY MORE, and if you remember one, that
+    memory is of a tautology (#3670). The search applies the bar before
+    returning, so every returned result cleared it by construction and a call
+    with no results has no score to compare: the field was true exactly when
+    `result_count > 0`, i.e. it was `calls - zero_result_calls` under a name
+    that promised a second opinion. `zero_result_calls + cleared_threshold ==
+    calls` held on all nineteen readings ever taken. The reading procedure
+    built on it — "clears its bar on nearly every call" — asked you to compare
+    a number with itself.
+
+    CHECK `suppression` BEFORE CONCLUDING ANYTHING FROM `zero_result_calls`. A
+    zero-result call is two different events wearing one number: the ranker
+    found nothing above the bar, or it found only what this session had already
+    been shown. Just the first is evidence about the bar. `suppression` splits
+    them where the surface can tell — `zero_because_already_shown` comes off
     `zero_result_calls` to leave the true ranker declines.
 
     `suppression` is `null` when NO row in the window reported it, and that is