fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/services/retrieval_telemetry.py

@@ -56,6 +56,7 @@ def _build_payload(
     results: list[tuple[float, Note]],
     duration_ms: float | None,
     suppressed: int | None = None,
+    best_available: float | None = None,
 ) -> dict:
     """Reduce a retrieval call to a flat, JSON-safe RetrievalLog payload.
 
@@ -67,6 +68,13 @@ def _build_payload(
     had already been shown them, and it stays None for callers that cannot
     know. See the column's comment: None means "not measured here", which is a
     different fact from 0 and must never render as one.
+
+    `best_available` is the highest score the ranker reached BEFORE the
+    threshold, and it carries the same null discipline for a sharper reason: it
+    is the only field that still says something on a call that returned
+    nothing, so a 0.0 standing in for "not measured" would read as "the corpus
+    held nothing remotely relevant" — a claim about the corpus invented out of
+    a caller's silence.
     """
     items = [
         {"id": int(note.id), "score": round(float(score), 5), "rank": rank}
@@ -85,6 +93,9 @@ def _build_payload(
         "suppressed_count": (None if suppressed is None else int(suppressed)),
         "top_score": (scores[0] if scores else None),
         "min_score": (scores[-1] if scores else None),
+        "best_available_score": (
+            None if best_available is None else round(float(best_available), 5)
+        ),
         "result_ids": items,
         "duration_ms": (round(duration_ms, 2) if duration_ms is not None else None),
     }
@@ -123,6 +134,7 @@ def record_retrieval(
     results: list[tuple[float, Any]],
     duration_ms: float | None = None,
     suppressed: int | None = None,
+    best_available: float | None = None,
 ) -> None:
     """Fire-and-forget: record one retrieval call.
 
@@ -149,6 +161,7 @@ def record_retrieval(
             results=results,
             duration_ms=duration_ms,
             suppressed=suppressed,
+            best_available=best_available,
         )
     except Exception:
         logger.debug("retrieval telemetry payload build failed", exc_info=True)
@@ -175,19 +188,25 @@ def record_retrieval(
 
 def _bucket(rows: list) -> dict:
     """A score readout a human can act on, from one aggregate row."""
-    (calls, zero, cleared, p10, p50, p90, lo, hi, avg_n, dur,
-     measured, supp_calls, supp_zero) = rows
+    (calls, zero, p10, p50, p90, lo, hi, avg_n, dur,
+     measured, supp_calls, supp_zero,
+     miss_calls, miss_p50, miss_p90, miss_max) = rows
     return {
         "calls": int(calls or 0),
         # A call that returned nothing is not a low-scoring call — it is a
         # different failure (nothing indexed, filter too narrow), and averaging
         # it into the score distribution would hide both.
         "zero_result_calls": int(zero or 0),
-        # How often the best hit actually cleared the threshold in force for
-        # that call. THE precision-adjacent number: a surface that clears its
-        # bar on almost every call is either well-tuned or too loose, and the
-        # score spread below says which.
-        "cleared_threshold": int(cleared or 0),
+        # `cleared_threshold` USED TO LIVE HERE and it was a tautology (#3670).
+        # The search applies the bar before returning, so every returned result
+        # cleared it by construction and a call with nothing has no score to
+        # compare — the condition was true exactly when `result_count > 0`.
+        # `zero_result_calls + cleared_threshold == calls` held on all nineteen
+        # readings ever taken. It was `calls - zero_result_calls` wearing a name
+        # that promised a second opinion, and the docstring built a reading
+        # procedure on it that asked the reader to compare a number with itself.
+        # Its replacement is `near_misses` below, which the bar cannot fix by
+        # construction because it is measured on the calls the bar REJECTED.
         # Of the zeros above, which were the RANKER declining and which were
         # the reader having seen it already? `zero_result_calls` cannot say,
         # and only the first kind is evidence about the threshold.
@@ -208,6 +227,28 @@ def _bucket(rows: list) -> dict:
             "p10": _round(p10), "p50": _round(p50), "p90": _round(p90),
             "min": _round(lo), "max": _round(hi),
         },
+        # WHAT THE BAR TURNED AWAY, and the only figure here a threshold can
+        # actually be tuned from. Measured over the calls that returned
+        # NOTHING, on the best score the ranker reached before the filter.
+        #
+        # Read `p90` against the threshold in force. A bar at 0.72 rejecting a
+        # stream of 0.71s is set too high by a hair and the surface is losing
+        # hits it should have had; the same bar rejecting 0.30s is doing its
+        # job and the corpus simply had nothing. Both render as a zero-result
+        # call, and nothing else in this readout separates them.
+        #
+        # None — not a zeroed block — when no declining call in the window
+        # measured it. Old rows predate the column, and a 0.0 would assert that
+        # the corpus held nothing relevant, which is a claim about the corpus
+        # invented out of a caller's silence.
+        "near_misses": (
+            None if not int(miss_calls or 0) else {
+                "measured_calls": int(miss_calls or 0),
+                "p50": _round(miss_p50),
+                "p90": _round(miss_p90),
+                "max": _round(miss_max),
+            }
+        ),
         "avg_result_count": _round(avg_n),
         "p90_duration_ms": _round(dur, 1),
     }
@@ -215,11 +256,13 @@ def _bucket(rows: list) -> dict:
 
 # The aggregate row Postgres would have returned for a source with no rows in
 # the window: nothing counted, nothing scored. Positional, matching the SELECT
-# `_bucket` unpacks — calls, zero, cleared, p10, p50, p90, min, max, avg_n,
-# dur, measured, supp_calls, supp_zero. The three counts are 0 because zero
-# calls is a real observation; everything else is None because a distribution
-# nobody sampled has no value, and rendering it as 0.0 would state one.
-_NO_ROWS_IN_WINDOW = [0, 0, 0, None, None, None, None, None, None, None, 0, 0, 0]
+# `_bucket` unpacks — calls, zero, p10, p50, p90, min, max, avg_n,
+# dur, measured, supp_calls, supp_zero, miss_calls, miss_p50, miss_p90,
+# miss_max. The counts are 0 because zero calls is a real observation;
+# everything else is None because a distribution nobody sampled has no value,
+# and rendering it as 0.0 would state one.
+_NO_ROWS_IN_WINDOW = [0, 0, None, None, None, None, None, None, None,
+                      0, 0, 0, 0, None, None, None]
 
 
 def _round(v, places: int = 4):
@@ -316,16 +359,22 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
         "read_failed": False,
     }
 
-    cleared = case(
-        (
-            (RetrievalLog.threshold.isnot(None))
-            & (RetrievalLog.top_score.isnot(None))
-            & (RetrievalLog.top_score >= RetrievalLog.threshold),
-            1,
-        ),
-        else_=0,
-    )
     zero = case((RetrievalLog.result_count == 0, 1), else_=0)
+    # THE NEAR-MISS POPULATION: calls that returned nothing AND recorded what
+    # the bar turned away. Both conditions matter. Restricting to zero-result
+    # calls is what makes the number say something the bar cannot fix by
+    # construction — on a call that returned something, `best_available_score`
+    # equals `top_score` and adds nothing. Requiring the column to be non-null
+    # keeps rows written before #3670 out of the sample rather than letting
+    # them read as scoreless declines.
+    declined = (RetrievalLog.result_count == 0) & (
+        RetrievalLog.best_available_score.isnot(None)
+    )
+    miss = case((declined, 1), else_=0)
+    # `best_available_score` only for those rows; NULL elsewhere, and
+    # percentile_cont ignores NULLs, so the distribution is over the declines
+    # alone without a second pass over the table.
+    miss_score = case((declined, RetrievalLog.best_available_score), else_=None)
     # Three sums rather than one, because "not measured" and "measured as zero"
     # are different answers and a single counter cannot hold both.
     measured = case((RetrievalLog.suppressed_count.isnot(None), 1), else_=0)
@@ -355,7 +404,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
                         RetrievalLog.source,
                         func.count().label("calls"),
                         func.sum(zero).label("zero"),
-                        func.sum(cleared).label("cleared"),
                         pct(0.1), pct(0.5), pct(0.9),
                         func.min(RetrievalLog.top_score),
                         func.max(RetrievalLog.top_score),
@@ -366,6 +414,10 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
                         func.sum(measured).label("measured"),
                         func.sum(supp_calls).label("supp_calls"),
                         func.sum(supp_zero).label("supp_zero"),
+                        func.sum(miss).label("miss_calls"),
+                        func.percentile_cont(0.5).within_group(miss_score.asc()),
+                        func.percentile_cont(0.9).within_group(miss_score.asc()),
+                        func.max(miss_score),
                     )
                     .where(
                         RetrievalLog.created_at >= since,