fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/services/embeddings.py

@@ -448,6 +448,23 @@ async def upsert_note_embedding(
         logger.warning("Failed to persist embedding for note %d", note_id, exc_info=True)
 
 
+# Both searches rank WITHOUT the threshold and apply it in Python, so the best
+# rejected score stays observable (#3670). The qualifying set is provably
+# unchanged: rows arrive ordered by distance ascending, so every above-bar row
+# sorts ahead of every below-bar one, and an over-fetch that used to return N
+# above-bar rows returns the same N plus some losers. What changes is only that
+# the losers are now visible instead of discarded inside the query.
+#
+# That visibility is the entire point. A bar can only be judged from the calls
+# it TURNED AWAY — a 0.72 bar rejecting a stream of 0.71s is set too high by a
+# hair, one rejecting 0.30s is working — and those two are indistinguishable
+# from any arrangement of the columns that survive the filter.
+#
+# `report` is how the score gets out without changing what a search RETURNS.
+# Eight of the eleven call sites want hits and nothing else; the three that
+# write telemetry pass a dict and read `best_available_score` back out of it.
+
+
 async def semantic_search_notes(
     user_id: int,
     query: str,
@@ -462,12 +479,19 @@ async def semantic_search_notes(
     scope: str = "own",
     demote_superseded: bool = True,
     system_id: int | None = None,
+    report: dict | None = None,
 ) -> list[tuple[float, Note]]:
     """Return up to *limit* (score, note) pairs most relevant to *query*.
 
     Scores are cosine similarities in [-1, 1]; only notes at or above
     *threshold* are returned, sorted highest-first.
 
+    Pass `report` (an empty dict) to learn what the threshold turned away:
+    the function sets `report["best_available_score"]` to the highest score
+    anything reached, or None when the corpus offered nothing at all. It is
+    the only figure that survives a call returning nothing, and therefore the
+    only one a bar can be judged from (#3670).
+
     `note_type` narrows to a record kind, or several (e.g. "snippet", or
     ("snippet", "note")), for callers that want prior art rather than everything
     embedded.
@@ -513,7 +537,6 @@ async def semantic_search_notes(
     # Distance ceiling equivalent to the similarity floor. Clamp to the valid
     # cosine-distance range [0, 2] so a threshold of, say, -1 doesn't produce a
     # nonsensical ceiling.
-    max_distance = min(2.0, max(0.0, 1.0 - threshold))
     distance = NoteEmbedding.embedding.cosine_distance(query_vec)
 
     try:
@@ -588,11 +611,10 @@ async def semantic_search_notes(
             fetch = limit * _CHUNK_OVERFETCH * (
                 _SUPERSESSION_OVERFETCH if demote_superseded else 1
             )
-            stmt = (
-                stmt.where(distance <= max_distance)
-                .order_by(distance.asc())
-                .limit(fetch)
-            )
+            # NO threshold predicate — see the note above this function. The
+            # bar is applied after the collapse, where the rejected scores can
+            # still be seen.
+            stmt = stmt.order_by(distance.asc()).limit(fetch)
             rows = list((await session.execute(stmt)).all())
     except Exception:
         logger.warning("Failed to query note embeddings", exc_info=True)
@@ -611,6 +633,11 @@ async def semantic_search_notes(
             continue
         seen.add(int(note.id))
         scored.append((1.0 - float(dist), note))
+    # The best score anything reached, bar or no bar. Recorded BEFORE the
+    # filter because a call that returns nothing is exactly when it matters.
+    if report is not None:
+        report["best_available_score"] = scored[0][0] if scored else None
+    scored = [pair for pair in scored if pair[0] >= threshold]
     if not demote_superseded:
         return scored[:limit]
     return await _apply_supersession_penalty(scored, limit)
@@ -764,9 +791,16 @@ async def semantic_search_rules(
     limit: int = 5,
     threshold: float = _SIMILARITY_THRESHOLD,
     tier: str | None = None,
+    report: dict | None = None,
 ) -> list[tuple[float, "Rule"]]:
     """Return up to *limit* (score, rule) pairs most relevant to *query*.
 
+    Pass `report` (an empty dict) to learn what the threshold turned away:
+    the function sets `report["best_available_score"]` to the highest score
+    anything reached, or None when the corpus offered nothing at all. It is
+    the only figure that survives a call returning nothing, and therefore the
+    only one a bar can be judged from (#3670).
+
     Scoped by OWNERSHIP — a rule is the caller's if they own its rulebook or
     its project. Deliberately not filtered to what currently BINDS a given
     project: this answers "is there a rule about this", which a person asking
@@ -802,7 +836,6 @@ async def semantic_search_rules(
         logger.debug("Rule search skipped — embedder unavailable")
         return []
 
-    max_distance = min(2.0, max(0.0, 1.0 - threshold))
     distance = RuleEmbedding.embedding.cosine_distance(query_vec)
 
     try:
@@ -816,7 +849,8 @@ async def semantic_search_rules(
                 .outerjoin(Project, Rule.project_id == Project.id)
                 .where(
                     Rule.deleted_at.is_(None),
-                    distance <= max_distance,
+                    # No threshold predicate — see the note above
+                    # semantic_search_notes. Applied below, after the collapse.
                     # topic_id XOR project_id, so exactly one arm can match.
                     or_(
                         Rulebook.owner_user_id == user_id,
@@ -839,7 +873,9 @@ async def semantic_search_rules(
         if rule.id not in best or score > best[rule.id][0]:
             best[rule.id] = (score, rule)
     ranked = sorted(best.values(), key=lambda pair: pair[0], reverse=True)
-    return ranked[:limit]
+    if report is not None:
+        report["best_available_score"] = ranked[0][0] if ranked else None
+    return [pair for pair in ranked if pair[0] >= threshold][:limit]
 
 
 async def backfill_rule_embeddings() -> None: