fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/mcp/tools/search.py

@@ -112,6 +112,7 @@ async def search(
         return await _search_rules(uid, q, limit)
     is_task = {"note": False, "task": True}.get(content_type)  # None => any
     t0 = time.perf_counter()
+    report: dict = {}
     raw = await semantic_search_notes(
         uid, q, limit=limit, is_task=is_task,
         project_id=project_id or None,
@@ -119,12 +120,14 @@ async def search(
         # An explicit search reaches everything the operator may read, including
         # records shared with them one-to-one.
         scope="read",
+        report=report,
     )
     record_retrieval(
         user_id=uid, source="mcp_search", query=q,
         threshold=DEFAULT_SIMILARITY_THRESHOLD, limit=limit,
         project_id=project_id or None, is_task=is_task, results=raw,
         duration_ms=(time.perf_counter() - t0) * 1000.0,
+        best_available=report.get("best_available_score"),
     )
     owners = await owner_names_for(
         {int(note.user_id) for _s, note in raw if note.user_id != uid}
@@ -162,18 +165,39 @@ async def retrieval_telemetry(days: int = 30) -> dict:
 
     `sources` — per retrieval surface (`auto_inject`, `write_path`,
     `mcp_search`, …), from `retrieval_logs`: `calls`, `zero_result_calls`,
-    `cleared_threshold` (how often the best hit beat the threshold in force for
-    that call), the `top_score` spread (p10/p50/p90/min/max), `avg_result_count`
-    and `p90_duration_ms`. THE number to read first is `cleared_threshold`
-    against `calls`, with the spread beside it: a surface that clears its bar
-    on nearly every call is either well-tuned or too loose, and p10 says which.
+    `near_misses`, the `top_score` spread (p10/p50/p90/min/max),
+    `avg_result_count` and `p90_duration_ms`.
 
-    READ `cleared_threshold` AND `zero_result_calls` TOGETHER, and check
-    `suppression` before concluding anything from either. A zero-result call is
-    two different events wearing one number: the ranker found nothing above the
-    bar, or it found only what this session had already been shown. Just the
-    first is evidence the bar is too high. `suppression` splits them where the
-    surface can tell — `zero_because_already_shown` comes off
+    THE NUMBER TO READ FIRST IS `near_misses.p90`, AGAINST THE THRESHOLD IN
+    FORCE FOR THAT SURFACE. It is measured only on the calls that returned
+    NOTHING, on the best score the ranker reached before the bar rejected it —
+    so it is the one figure here that says something the bar cannot make true
+    by construction. A bar at 0.72 turning away a stream of 0.71s is set too
+    high by a hair and the surface is losing hits it should have had. The same
+    bar turning away 0.30s is working, and the corpus simply had nothing. Both
+    render as a zero-result call, and nothing else in this readout tells them
+    apart.
+
+    `near_misses` is `null` when no declining call in the window measured it —
+    rows written before #3670 shipped cannot know. That is "not measured", not
+    "nothing came close"; a 0.0 there would be a claim about the corpus
+    invented out of a caller's silence.
+
+    THERE IS NO `cleared_threshold` ANY MORE, and if you remember one, that
+    memory is of a tautology (#3670). The search applies the bar before
+    returning, so every returned result cleared it by construction and a call
+    with no results has no score to compare: the field was true exactly when
+    `result_count > 0`, i.e. it was `calls - zero_result_calls` under a name
+    that promised a second opinion. `zero_result_calls + cleared_threshold ==
+    calls` held on all nineteen readings ever taken. The reading procedure
+    built on it — "clears its bar on nearly every call" — asked you to compare
+    a number with itself.
+
+    CHECK `suppression` BEFORE CONCLUDING ANYTHING FROM `zero_result_calls`. A
+    zero-result call is two different events wearing one number: the ranker
+    found nothing above the bar, or it found only what this session had already
+    been shown. Just the first is evidence about the bar. `suppression` splits
+    them where the surface can tell — `zero_because_already_shown` comes off
     `zero_result_calls` to leave the true ranker declines.
 
     `suppression` is `null` when NO row in the window reported it, and that is

src/scribe/models/retrieval_log.py

@@ -54,6 +54,14 @@ class RetrievalLog(Base):
     suppressed_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
     top_score: Mapped[float | None] = mapped_column(Float, nullable=True)
     min_score: Mapped[float | None] = mapped_column(Float, nullable=True)
+    # The best score the ranker COULD have offered, before the threshold — as
+    # against `top_score`, which is the best it DID offer. They are equal on
+    # any call that returned something, and only this one exists on a call
+    # that returned nothing, which is the only place a bar can be judged from
+    # (#3670). Null means the caller did not measure it, never "nothing was
+    # close": a 0.0 there would read as a corpus with no relevant records at
+    # all, which is an artifact standing in for a measurement.
+    best_available_score: Mapped[float | None] = mapped_column(Float, nullable=True)
     # [{"id": int, "score": float, "rank": int}, ...], highest-first.
     result_ids: Mapped[list] = mapped_column(JSONB, nullable=False, default=list)
     duration_ms: Mapped[float | None] = mapped_column(Float, nullable=True)

src/scribe/routes/search.py

@@ -44,17 +44,20 @@ async def search_route():
     project_id = request.args.get("project_id", type=int)
 
     t0 = time.perf_counter()
+    report: dict = {}
     results = await semantic_search_notes(
         uid, q, limit=limit, is_task=is_task, threshold=_REST_SEARCH_THRESHOLD,
         project_id=project_id, system_id=system_id,
         # The user typed this, so it reaches everything they may read.
         scope="read",
+        report=report,
     )
     record_retrieval(
         user_id=uid, source="rest_search", query=q,
         threshold=_REST_SEARCH_THRESHOLD, limit=limit,
         project_id=project_id, is_task=is_task, results=results,
         duration_ms=(time.perf_counter() - t0) * 1000.0,
+        best_available=report.get("best_available_score"),
     )
     owners = await owner_names_for(
         {int(note.user_id) for _s, note in results if note.user_id != uid}

src/scribe/services/embeddings.py

@@ -448,6 +448,23 @@ async def upsert_note_embedding(
         logger.warning("Failed to persist embedding for note %d", note_id, exc_info=True)
 
 
+# Both searches rank WITHOUT the threshold and apply it in Python, so the best
+# rejected score stays observable (#3670). The qualifying set is provably
+# unchanged: rows arrive ordered by distance ascending, so every above-bar row
+# sorts ahead of every below-bar one, and an over-fetch that used to return N
+# above-bar rows returns the same N plus some losers. What changes is only that
+# the losers are now visible instead of discarded inside the query.
+#
+# That visibility is the entire point. A bar can only be judged from the calls
+# it TURNED AWAY — a 0.72 bar rejecting a stream of 0.71s is set too high by a
+# hair, one rejecting 0.30s is working — and those two are indistinguishable
+# from any arrangement of the columns that survive the filter.
+#
+# `report` is how the score gets out without changing what a search RETURNS.
+# Eight of the eleven call sites want hits and nothing else; the three that
+# write telemetry pass a dict and read `best_available_score` back out of it.
+
+
 async def semantic_search_notes(
     user_id: int,
     query: str,
@@ -462,12 +479,19 @@ async def semantic_search_notes(
     scope: str = "own",
     demote_superseded: bool = True,
     system_id: int | None = None,
+    report: dict | None = None,
 ) -> list[tuple[float, Note]]:
     """Return up to *limit* (score, note) pairs most relevant to *query*.
 
     Scores are cosine similarities in [-1, 1]; only notes at or above
     *threshold* are returned, sorted highest-first.
 
+    Pass `report` (an empty dict) to learn what the threshold turned away:
+    the function sets `report["best_available_score"]` to the highest score
+    anything reached, or None when the corpus offered nothing at all. It is
+    the only figure that survives a call returning nothing, and therefore the
+    only one a bar can be judged from (#3670).
+
     `note_type` narrows to a record kind, or several (e.g. "snippet", or
     ("snippet", "note")), for callers that want prior art rather than everything
     embedded.
@@ -513,7 +537,6 @@ async def semantic_search_notes(
     # Distance ceiling equivalent to the similarity floor. Clamp to the valid
     # cosine-distance range [0, 2] so a threshold of, say, -1 doesn't produce a
     # nonsensical ceiling.
-    max_distance = min(2.0, max(0.0, 1.0 - threshold))
     distance = NoteEmbedding.embedding.cosine_distance(query_vec)
 
     try:
@@ -588,11 +611,10 @@ async def semantic_search_notes(
             fetch = limit * _CHUNK_OVERFETCH * (
                 _SUPERSESSION_OVERFETCH if demote_superseded else 1
             )
-            stmt = (
-                stmt.where(distance <= max_distance)
-                .order_by(distance.asc())
-                .limit(fetch)
-            )
+            # NO threshold predicate — see the note above this function. The
+            # bar is applied after the collapse, where the rejected scores can
+            # still be seen.
+            stmt = stmt.order_by(distance.asc()).limit(fetch)
             rows = list((await session.execute(stmt)).all())
     except Exception:
         logger.warning("Failed to query note embeddings", exc_info=True)
@@ -611,6 +633,11 @@ async def semantic_search_notes(
             continue
         seen.add(int(note.id))
         scored.append((1.0 - float(dist), note))
+    # The best score anything reached, bar or no bar. Recorded BEFORE the
+    # filter because a call that returns nothing is exactly when it matters.
+    if report is not None:
+        report["best_available_score"] = scored[0][0] if scored else None
+    scored = [pair for pair in scored if pair[0] >= threshold]
     if not demote_superseded:
         return scored[:limit]
     return await _apply_supersession_penalty(scored, limit)
@@ -764,9 +791,16 @@ async def semantic_search_rules(
     limit: int = 5,
     threshold: float = _SIMILARITY_THRESHOLD,
     tier: str | None = None,
+    report: dict | None = None,
 ) -> list[tuple[float, "Rule"]]:
     """Return up to *limit* (score, rule) pairs most relevant to *query*.
 
+    Pass `report` (an empty dict) to learn what the threshold turned away:
+    the function sets `report["best_available_score"]` to the highest score
+    anything reached, or None when the corpus offered nothing at all. It is
+    the only figure that survives a call returning nothing, and therefore the
+    only one a bar can be judged from (#3670).
+
     Scoped by OWNERSHIP — a rule is the caller's if they own its rulebook or
     its project. Deliberately not filtered to what currently BINDS a given
     project: this answers "is there a rule about this", which a person asking
@@ -802,7 +836,6 @@ async def semantic_search_rules(
         logger.debug("Rule search skipped — embedder unavailable")
         return []
 
-    max_distance = min(2.0, max(0.0, 1.0 - threshold))
     distance = RuleEmbedding.embedding.cosine_distance(query_vec)
 
     try:
@@ -816,7 +849,8 @@ async def semantic_search_rules(
                 .outerjoin(Project, Rule.project_id == Project.id)
                 .where(
                     Rule.deleted_at.is_(None),
-                    distance <= max_distance,
+                    # No threshold predicate — see the note above
+                    # semantic_search_notes. Applied below, after the collapse.
                     # topic_id XOR project_id, so exactly one arm can match.
                     or_(
                         Rulebook.owner_user_id == user_id,
@@ -839,7 +873,9 @@ async def semantic_search_rules(
         if rule.id not in best or score > best[rule.id][0]:
             best[rule.id] = (score, rule)
     ranked = sorted(best.values(), key=lambda pair: pair[0], reverse=True)
-    return ranked[:limit]
+    if report is not None:
+        report["best_available_score"] = ranked[0][0] if ranked else None
+    return [pair for pair in ranked if pair[0] >= threshold][:limit]
 
 
 async def backfill_rule_embeddings() -> None:

src/scribe/services/plugin_context.py

@@ -463,6 +463,7 @@ async def _reserve_slot_for_reuse(
 
     top_k = cfg["top_k"]
     _t0 = time.perf_counter()
+    _rep: dict = {}
     reuse = await semantic_search_notes(
         user_id, query,
         limit=1,
@@ -471,6 +472,7 @@ async def _reserve_slot_for_reuse(
         exclude_ids=exclude_ids | {int(n.id) for _s, n in kept},
         note_type=_REUSE_KINDS,
         scope="browse",
+        report=_rep,
     )
     # A real semantic query competing for a menu slot — logged like the scored
     # arm it displaces. Before this, the hit it PUSHED OUT was in
@@ -481,6 +483,7 @@ async def _reserve_slot_for_reuse(
         user_id=user_id, source="reuse_slot", query=query,
         threshold=cfg["threshold"], limit=1, project_id=project_id,
         is_task=None, results=reuse,
+        best_available=_rep.get("best_available_score"),
         duration_ms=(time.perf_counter() - _t0) * 1000.0,
     )
     # Verify the kind rather than trusting the query that asked for it, and
@@ -530,6 +533,7 @@ async def build_autoinject_hint(
         return empty
 
     t0 = time.perf_counter()
+    _rep_ai: dict = {}
     hits = await semantic_search_notes(
         user_id, q,
         limit=cfg["top_k"],
@@ -541,11 +545,13 @@ async def build_autoinject_hint(
         # still appear is a collaborator's note inside a shared project — legible
         # only because the line below names its owner.
         scope="browse",
+        report=_rep_ai,
     )
     record_retrieval(
         user_id=user_id, source="auto_inject", query=q,
         threshold=cfg["threshold"], limit=cfg["top_k"],
         project_id=(project_id or None), is_task=None, results=hits,
+        best_available=_rep_ai.get("best_available_score"),
         duration_ms=(time.perf_counter() - t0) * 1000.0,
     )
     if not hits:
@@ -972,6 +978,7 @@ async def build_write_path_hint(
         # Pulled-and-seen ids stay in the query (as evidence) but never in
         # the menu — the dedup contract holds, the resemblance still lands.
         pulled_seen = seen & set(pulled)
+        _rep_wp: dict = {}
         hits = await semantic_search_notes(
             user_id, query,
             limit=remaining + len(pulled_seen),
@@ -995,6 +1002,7 @@ async def build_write_path_hint(
             # Same reasoning as auto-inject: nobody asked for this, so it takes
             # the browse scope and never surfaces a one-to-one direct share.
             scope="browse",
+            report=_rep_wp,
         )
         resembles = {
             int(note.id): float(score) for score, note in hits
@@ -1008,6 +1016,7 @@ async def build_write_path_hint(
             # recording it as a notes-only retrieval would misdescribe the
             # candidate set the threshold is being tuned against.
             project_id=scope_project, is_task=None, results=hits,
+            best_available=_rep_wp.get("best_available_score"),
             duration_ms=(time.perf_counter() - t0) * 1000.0,
         )
         if hits:
@@ -1251,9 +1260,11 @@ async def build_write_path_hint(
         # — a gap that reads as "this surface is somehow not measurable" rather
         # than "nobody passed the number".
         rule_t0 = time.perf_counter()
+        _rep_wpr: dict = {}
         hits = await semantic_search_rules(
             user_id, code or path, limit=RULEHINT_LIMIT,
             threshold=cfg["rule_threshold"],
+            report=_rep_wpr,
         )
         rule_ms = (time.perf_counter() - rule_t0) * 1000.0
         fresh = [(score, rule) for score, rule in hits if rule.id not in already]
@@ -1301,6 +1312,7 @@ async def build_write_path_hint(
             threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
             project_id=project_id,
             is_task=None, results=fresh, duration_ms=rule_ms,
+            best_available=_rep_wpr.get("best_available_score"),
             # What the ranker found and this session had already been told.
             # Without it a zero row cannot say whether the bar was too high or
             # the reader was simply ahead of it — and only the first is a
@@ -1383,9 +1395,11 @@ async def build_tool_rule_hint(
         query = command[:_TOOL_QUERY_CHARS]
 
         t0 = time.perf_counter()
+        _rep_ptr: dict = {}
         hits = await semantic_search_rules(
             user_id, query, limit=RULEHINT_LIMIT,
             threshold=cfg["rule_threshold"],
+            report=_rep_ptr,
         )
         duration_ms = (time.perf_counter() - t0) * 1000.0
 
@@ -1405,6 +1419,7 @@ async def build_tool_rule_hint(
             threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
             project_id=project_id,
             is_task=None, results=fresh, duration_ms=duration_ms,
+            best_available=_rep_ptr.get("best_available_score"),
             # See the sibling arm. It matters more here: this arm fires on every
             # Bash call, so a long session excludes its way to an all-zero row
             # and the threshold looks wrong when nothing about it is.

src/scribe/services/retrieval_telemetry.py

@@ -56,6 +56,7 @@ def _build_payload(
     results: list[tuple[float, Note]],
     duration_ms: float | None,
     suppressed: int | None = None,
+    best_available: float | None = None,
 ) -> dict:
     """Reduce a retrieval call to a flat, JSON-safe RetrievalLog payload.
 
@@ -67,6 +68,13 @@ def _build_payload(
     had already been shown them, and it stays None for callers that cannot
     know. See the column's comment: None means "not measured here", which is a
     different fact from 0 and must never render as one.
+
+    `best_available` is the highest score the ranker reached BEFORE the
+    threshold, and it carries the same null discipline for a sharper reason: it
+    is the only field that still says something on a call that returned
+    nothing, so a 0.0 standing in for "not measured" would read as "the corpus
+    held nothing remotely relevant" — a claim about the corpus invented out of
+    a caller's silence.
     """
     items = [
         {"id": int(note.id), "score": round(float(score), 5), "rank": rank}
@@ -85,6 +93,9 @@ def _build_payload(
         "suppressed_count": (None if suppressed is None else int(suppressed)),
         "top_score": (scores[0] if scores else None),
         "min_score": (scores[-1] if scores else None),
+        "best_available_score": (
+            None if best_available is None else round(float(best_available), 5)
+        ),
         "result_ids": items,
         "duration_ms": (round(duration_ms, 2) if duration_ms is not None else None),
     }
@@ -123,6 +134,7 @@ def record_retrieval(
     results: list[tuple[float, Any]],
     duration_ms: float | None = None,
     suppressed: int | None = None,
+    best_available: float | None = None,
 ) -> None:
     """Fire-and-forget: record one retrieval call.
 
@@ -149,6 +161,7 @@ def record_retrieval(
             results=results,
             duration_ms=duration_ms,
             suppressed=suppressed,
+            best_available=best_available,
         )
     except Exception:
         logger.debug("retrieval telemetry payload build failed", exc_info=True)
@@ -175,19 +188,25 @@ def record_retrieval(
 
 def _bucket(rows: list) -> dict:
     """A score readout a human can act on, from one aggregate row."""
-    (calls, zero, cleared, p10, p50, p90, lo, hi, avg_n, dur,
-     measured, supp_calls, supp_zero) = rows
+    (calls, zero, p10, p50, p90, lo, hi, avg_n, dur,
+     measured, supp_calls, supp_zero,
+     miss_calls, miss_p50, miss_p90, miss_max) = rows
     return {
         "calls": int(calls or 0),
         # A call that returned nothing is not a low-scoring call — it is a
         # different failure (nothing indexed, filter too narrow), and averaging
         # it into the score distribution would hide both.
         "zero_result_calls": int(zero or 0),
-        # How often the best hit actually cleared the threshold in force for
-        # that call. THE precision-adjacent number: a surface that clears its
-        # bar on almost every call is either well-tuned or too loose, and the
-        # score spread below says which.
-        "cleared_threshold": int(cleared or 0),
+        # `cleared_threshold` USED TO LIVE HERE and it was a tautology (#3670).
+        # The search applies the bar before returning, so every returned result
+        # cleared it by construction and a call with nothing has no score to
+        # compare — the condition was true exactly when `result_count > 0`.
+        # `zero_result_calls + cleared_threshold == calls` held on all nineteen
+        # readings ever taken. It was `calls - zero_result_calls` wearing a name
+        # that promised a second opinion, and the docstring built a reading
+        # procedure on it that asked the reader to compare a number with itself.
+        # Its replacement is `near_misses` below, which the bar cannot fix by
+        # construction because it is measured on the calls the bar REJECTED.
         # Of the zeros above, which were the RANKER declining and which were
         # the reader having seen it already? `zero_result_calls` cannot say,
         # and only the first kind is evidence about the threshold.
@@ -208,6 +227,28 @@ def _bucket(rows: list) -> dict:
             "p10": _round(p10), "p50": _round(p50), "p90": _round(p90),
             "min": _round(lo), "max": _round(hi),
         },
+        # WHAT THE BAR TURNED AWAY, and the only figure here a threshold can
+        # actually be tuned from. Measured over the calls that returned
+        # NOTHING, on the best score the ranker reached before the filter.
+        #
+        # Read `p90` against the threshold in force. A bar at 0.72 rejecting a
+        # stream of 0.71s is set too high by a hair and the surface is losing
+        # hits it should have had; the same bar rejecting 0.30s is doing its
+        # job and the corpus simply had nothing. Both render as a zero-result
+        # call, and nothing else in this readout separates them.
+        #
+        # None — not a zeroed block — when no declining call in the window
+        # measured it. Old rows predate the column, and a 0.0 would assert that
+        # the corpus held nothing relevant, which is a claim about the corpus
+        # invented out of a caller's silence.
+        "near_misses": (
+            None if not int(miss_calls or 0) else {
+                "measured_calls": int(miss_calls or 0),
+                "p50": _round(miss_p50),
+                "p90": _round(miss_p90),
+                "max": _round(miss_max),
+            }
+        ),
         "avg_result_count": _round(avg_n),
         "p90_duration_ms": _round(dur, 1),
     }
@@ -215,11 +256,13 @@ def _bucket(rows: list) -> dict:
 
 # The aggregate row Postgres would have returned for a source with no rows in
 # the window: nothing counted, nothing scored. Positional, matching the SELECT
-# `_bucket` unpacks — calls, zero, cleared, p10, p50, p90, min, max, avg_n,
-# dur, measured, supp_calls, supp_zero. The three counts are 0 because zero
-# calls is a real observation; everything else is None because a distribution
-# nobody sampled has no value, and rendering it as 0.0 would state one.
-_NO_ROWS_IN_WINDOW = [0, 0, 0, None, None, None, None, None, None, None, 0, 0, 0]
+# `_bucket` unpacks — calls, zero, p10, p50, p90, min, max, avg_n,
+# dur, measured, supp_calls, supp_zero, miss_calls, miss_p50, miss_p90,
+# miss_max. The counts are 0 because zero calls is a real observation;
+# everything else is None because a distribution nobody sampled has no value,
+# and rendering it as 0.0 would state one.
+_NO_ROWS_IN_WINDOW = [0, 0, None, None, None, None, None, None, None,
+                      0, 0, 0, 0, None, None, None]
 
 
 def _round(v, places: int = 4):
@@ -316,16 +359,22 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
         "read_failed": False,
     }
 
-    cleared = case(
-        (
-            (RetrievalLog.threshold.isnot(None))
-            & (RetrievalLog.top_score.isnot(None))
-            & (RetrievalLog.top_score >= RetrievalLog.threshold),
-            1,
-        ),
-        else_=0,
-    )
     zero = case((RetrievalLog.result_count == 0, 1), else_=0)
+    # THE NEAR-MISS POPULATION: calls that returned nothing AND recorded what
+    # the bar turned away. Both conditions matter. Restricting to zero-result
+    # calls is what makes the number say something the bar cannot fix by
+    # construction — on a call that returned something, `best_available_score`
+    # equals `top_score` and adds nothing. Requiring the column to be non-null
+    # keeps rows written before #3670 out of the sample rather than letting
+    # them read as scoreless declines.
+    declined = (RetrievalLog.result_count == 0) & (
+        RetrievalLog.best_available_score.isnot(None)
+    )
+    miss = case((declined, 1), else_=0)
+    # `best_available_score` only for those rows; NULL elsewhere, and
+    # percentile_cont ignores NULLs, so the distribution is over the declines
+    # alone without a second pass over the table.
+    miss_score = case((declined, RetrievalLog.best_available_score), else_=None)
     # Three sums rather than one, because "not measured" and "measured as zero"
     # are different answers and a single counter cannot hold both.
     measured = case((RetrievalLog.suppressed_count.isnot(None), 1), else_=0)
@@ -355,7 +404,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
                         RetrievalLog.source,
                         func.count().label("calls"),
                         func.sum(zero).label("zero"),
-                        func.sum(cleared).label("cleared"),
                         pct(0.1), pct(0.5), pct(0.9),
                         func.min(RetrievalLog.top_score),
                         func.max(RetrievalLog.top_score),
@@ -366,6 +414,10 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
                         func.sum(measured).label("measured"),
                         func.sum(supp_calls).label("supp_calls"),
                         func.sum(supp_zero).label("supp_zero"),
+                        func.sum(miss).label("miss_calls"),
+                        func.percentile_cont(0.5).within_group(miss_score.asc()),
+                        func.percentile_cont(0.9).within_group(miss_score.asc()),
+                        func.max(miss_score),
                     )
                     .where(
                         RetrievalLog.created_at >= since,