fix(telemetry): the bar can only be judged from what it rejected (#3670)

`cleared_threshold` was documented as the number to read first. It was a tautology. The search applies the threshold before returning, so every returned result cleared it by construction and a call with no results has no top_score to compare — the condition was true exactly when `result_count > 0`. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen source/window readings ever taken, today's live seven included. The reading procedure built on it asked the reader to compare a number with itself, and a threshold change was unobservable through it: raise the bar and both numbers move together, so the field could never show a bar set too high. REPLACED, NOT JUST REMOVED. The question the table exists to answer is whether the bar is in the right place, and that is only answerable from the calls that returned NOTHING: how close did the best rejected candidate come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair; the same bar turning away 0.30s is working. Both render as a zero-result call today and nothing separates them, because the losing score is discarded inside the search. So both searches now rank WITHOUT the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one, and an over-fetch that returned N above-bar rows returns the same N plus some losers. What changes is that the losers are visible instead of dropped in the query. `report` carries the score out without changing what a search RETURNS: eight of eleven call sites want hits and nothing else. New column (migration 0096), nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence, which is the substitution this whole milestone corrects. The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step from the shape that produced #2663, where a rejected query was swallowed by the broad except and every counter read zero. It carries an integration guard for that reason: only real Postgres can say it parses, and the symptom of failure is silence. Also adds a guard that no int field in a bucket equals `calls - zero_result_calls`. That identity is what `cleared_threshold` satisfied for its whole life, and it survived because it had its own name and nobody added the two numbers beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
2026-09-08 13:50:05 -04:00
parent 277aea58e4
commit e7c1af32a0
8 changed files with 425 additions and 49 deletions
@@ -0,0 +1,62 @@
+"""add retrieval_logs.best_available_score — the score the bar rejected (#3670)
+
+Revision ID: 0096
+Revises: 0095
+Create Date: 2026-09-08
+
+`cleared_threshold` was documented as the number to read FIRST — "a surface
+that clears its bar on nearly every call is either well-tuned or too loose,
+and p10 says which". It was never a measurement. The search applies the
+threshold before returning, so every returned result cleared the bar by
+construction and a call with no results has no `top_score` to compare:
+the condition is true exactly when `result_count > 0`.
+
+`zero_result_calls + cleared_threshold == calls` held on all nineteen
+source/window readings ever taken. It was `calls - zero_result_calls`
+wearing a name that promised a second opinion, and a reading procedure was
+built on top of it that asked the reader to compare a number against itself.
+
+THE MISSING NUMBER, and the reason this is a column rather than a deletion.
+The question the table exists to answer is "is the bar in the right place",
+and that question is only answerable from the calls that returned NOTHING:
+how close did the best rejected candidate come? A bar at 0.72 turning away
+a stream of 0.71s is set too high by a hair. A bar turning away 0.30s is
+doing its job. Those two are indistinguishable today — both render as a
+zero-result call — and no arrangement of the existing columns separates
+them, because the losing score is discarded inside the search.
+
+So the searches now rank without the bar and apply it in Python, which
+costs nothing (the rows were already ordered by distance, and the qualifying
+set is provably identical — above-threshold rows sort first), and the best
+score seen becomes observable.
+
+NULLABLE, AND UNBACKFILLED, for the reason 0095 spells out: a row written
+before this shipped genuinely does not know what its best rejected candidate
+scored, and saying so is the honest state. A 0.0 default would read as "the
+corpus had nothing remotely relevant" — an artifact standing in for a
+measurement, which is the whole defect this milestone corrects.
+
+`retrieval_logs` is not restored from backup, so no importer changes.
+
+Downgrade drops the column. Purely observational — nothing reads it for
+correctness.
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+revision = "0096"
+down_revision = "0095"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "retrieval_logs",
+        sa.Column("best_available_score", sa.Float(), nullable=True),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("retrieval_logs", "best_available_score")