fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/services/plugin_context.py

@@ -463,6 +463,7 @@ async def _reserve_slot_for_reuse(
 
     top_k = cfg["top_k"]
     _t0 = time.perf_counter()
+    _rep: dict = {}
     reuse = await semantic_search_notes(
         user_id, query,
         limit=1,
@@ -471,6 +472,7 @@ async def _reserve_slot_for_reuse(
         exclude_ids=exclude_ids | {int(n.id) for _s, n in kept},
         note_type=_REUSE_KINDS,
         scope="browse",
+        report=_rep,
     )
     # A real semantic query competing for a menu slot — logged like the scored
     # arm it displaces. Before this, the hit it PUSHED OUT was in
@@ -481,6 +483,7 @@ async def _reserve_slot_for_reuse(
         user_id=user_id, source="reuse_slot", query=query,
         threshold=cfg["threshold"], limit=1, project_id=project_id,
         is_task=None, results=reuse,
+        best_available=_rep.get("best_available_score"),
         duration_ms=(time.perf_counter() - _t0) * 1000.0,
     )
     # Verify the kind rather than trusting the query that asked for it, and
@@ -530,6 +533,7 @@ async def build_autoinject_hint(
         return empty
 
     t0 = time.perf_counter()
+    _rep_ai: dict = {}
     hits = await semantic_search_notes(
         user_id, q,
         limit=cfg["top_k"],
@@ -541,11 +545,13 @@ async def build_autoinject_hint(
         # still appear is a collaborator's note inside a shared project — legible
         # only because the line below names its owner.
         scope="browse",
+        report=_rep_ai,
     )
     record_retrieval(
         user_id=user_id, source="auto_inject", query=q,
         threshold=cfg["threshold"], limit=cfg["top_k"],
         project_id=(project_id or None), is_task=None, results=hits,
+        best_available=_rep_ai.get("best_available_score"),
         duration_ms=(time.perf_counter() - t0) * 1000.0,
     )
     if not hits:
@@ -972,6 +978,7 @@ async def build_write_path_hint(
         # Pulled-and-seen ids stay in the query (as evidence) but never in
         # the menu — the dedup contract holds, the resemblance still lands.
         pulled_seen = seen & set(pulled)
+        _rep_wp: dict = {}
         hits = await semantic_search_notes(
             user_id, query,
             limit=remaining + len(pulled_seen),
@@ -995,6 +1002,7 @@ async def build_write_path_hint(
             # Same reasoning as auto-inject: nobody asked for this, so it takes
             # the browse scope and never surfaces a one-to-one direct share.
             scope="browse",
+            report=_rep_wp,
         )
         resembles = {
             int(note.id): float(score) for score, note in hits
@@ -1008,6 +1016,7 @@ async def build_write_path_hint(
             # recording it as a notes-only retrieval would misdescribe the
             # candidate set the threshold is being tuned against.
             project_id=scope_project, is_task=None, results=hits,
+            best_available=_rep_wp.get("best_available_score"),
             duration_ms=(time.perf_counter() - t0) * 1000.0,
         )
         if hits:
@@ -1251,9 +1260,11 @@ async def build_write_path_hint(
         # — a gap that reads as "this surface is somehow not measurable" rather
         # than "nobody passed the number".
         rule_t0 = time.perf_counter()
+        _rep_wpr: dict = {}
         hits = await semantic_search_rules(
             user_id, code or path, limit=RULEHINT_LIMIT,
             threshold=cfg["rule_threshold"],
+            report=_rep_wpr,
         )
         rule_ms = (time.perf_counter() - rule_t0) * 1000.0
         fresh = [(score, rule) for score, rule in hits if rule.id not in already]
@@ -1301,6 +1312,7 @@ async def build_write_path_hint(
             threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
             project_id=project_id,
             is_task=None, results=fresh, duration_ms=rule_ms,
+            best_available=_rep_wpr.get("best_available_score"),
             # What the ranker found and this session had already been told.
             # Without it a zero row cannot say whether the bar was too high or
             # the reader was simply ahead of it — and only the first is a
@@ -1383,9 +1395,11 @@ async def build_tool_rule_hint(
         query = command[:_TOOL_QUERY_CHARS]
 
         t0 = time.perf_counter()
+        _rep_ptr: dict = {}
         hits = await semantic_search_rules(
             user_id, query, limit=RULEHINT_LIMIT,
             threshold=cfg["rule_threshold"],
+            report=_rep_ptr,
         )
         duration_ms = (time.perf_counter() - t0) * 1000.0
 
@@ -1405,6 +1419,7 @@ async def build_tool_rule_hint(
             threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
             project_id=project_id,
             is_task=None, results=fresh, duration_ms=duration_ms,
+            best_available=_rep_ptr.get("best_available_score"),
             # See the sibling arm. It matters more here: this arm fires on every
             # Bash call, so a long session excludes its way to an all-zero row
             # and the threshold looks wrong when nothing about it is.