A search that never ran is not a decline #147

2026-09-09T13:36:37-04:00

bvandeusen commented

2026-09-09 13:36:37 -04:00

best_available_score was added by #3670 so a bar could be judged from what it rejected. It arrived null on four unrelated causes: the corpus offered nothing, the query was empty, the embedder was down, or the database query failed.

Only the first is a measurement. The fourth is the #2663 shape — a swallowed failure rendering as a clean zero — inside the field added to fix an instance of the #2663 shape.

How it surfaced

Trying to explain why reuse_slot returned nothing on 45 of 45 calls, and auto_inject on 153 of 161. That investigation is still open; what it established first is that the readout could not answer it. A retrieval problem and a broken query were spelled the same way.

The fix is not a new column

A call that never searched writes no row, so every remaining null means one thing: searched, and nothing came close.

That reuses the convention the pre-tool arm already follows for a blank command — "a row here would report a call that never happened and drag the clear-rate down with phantom declines" — extended from the case a caller can see in advance to the three only the search knows about.

The detail that did the real work

Both searches stamp report["searched"] False before anything can return, and True only where a real result set exists.

My first cut set it True on success only. Cleaner-looking, and wrong: 66 mocked searches across twelve test files never touch report, so every one would have read as "never searched" and stopped logging — a 66-site rewrite to simulate a flag those tests do not care about.

Stamping False up front makes absent a meaningful third state. Only a real search can report False, so absent means "no caller asked" and defaults to the old behaviour. Blast radius went from twelve files to zero, and the semantics got sharper rather than looser.

A failure is not made invisible

semantic_search_notes already logs a WARNING on query failure, which is where a broken search belongs. A counter cannot say "I am broken" without a reader already trusting it.

Three tests, and the middle one is what makes them discriminate

A blanket return passes "no row when unsearched" and fails "row when searched and empty" — and that second case is the only evidence a threshold is too high (#3497).

What this unblocks

After deploy, the reuse_slot question answers itself on the next read: rows still appearing with a null score mean searched, found nothing at any score; rows disappearing mean the searches were failing or being skipped. Either way, without reading the application log again.

CI green on 6234643 (run 6177), all six lanes. Server-side only — no plugin version change.

🤖 Generated with Claude Code

https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

`best_available_score` was added by #3670 so a bar could be judged from what it rejected. It arrived null on four unrelated causes: the corpus offered nothing, the query was empty, the embedder was down, or **the database query failed**. Only the first is a measurement. The fourth is the #2663 shape — a swallowed failure rendering as a clean zero — inside the field added to fix an instance of the #2663 shape. ## How it surfaced Trying to explain why `reuse_slot` returned nothing on 45 of 45 calls, and `auto_inject` on 153 of 161. That investigation is still open; what it established first is that **the readout could not answer it**. A retrieval problem and a broken query were spelled the same way. ## The fix is not a new column A call that never searched writes **no row**, so every remaining null means one thing: *searched, and nothing came close.* That reuses the convention the pre-tool arm already follows for a blank command — *"a row here would report a call that never happened and drag the clear-rate down with phantom declines"* — extended from the case a caller can see in advance to the three only the search knows about. ## The detail that did the real work Both searches stamp `report["searched"]` **False before anything can return**, and True only where a real result set exists. My first cut set it True on success only. Cleaner-looking, and wrong: 66 mocked searches across twelve test files never touch `report`, so every one would have read as "never searched" and stopped logging — a 66-site rewrite to simulate a flag those tests do not care about. Stamping False up front makes **absent** a meaningful third state. Only a real search can report False, so absent means "no caller asked" and defaults to the old behaviour. Blast radius went from twelve files to zero, and the semantics got sharper rather than looser. ## A failure is not made invisible `semantic_search_notes` already logs a WARNING on query failure, which is where a broken search belongs. A counter cannot say "I am broken" without a reader already trusting it. ## Three tests, and the middle one is what makes them discriminate A blanket `return` passes "no row when unsearched" and fails "row when searched and empty" — and that second case is the only evidence a threshold is too high (#3497). ## What this unblocks After deploy, the `reuse_slot` question answers itself on the next read: rows still appearing with a null score mean *searched, found nothing at any score*; rows disappearing mean the searches were failing or being skipped. Either way, without reading the application log again. CI green on `6234643` (run 6177), all six lanes. Server-side only — no plugin version change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

bvandeusen added 1 commit 2026-09-09 13:36:37 -04:00

fix(telemetry): a search that never ran is not a decline (#3765 )

CI & Build / Plugin hooks (push) Successful in 13s

Details

CI & Build / Python lint (push) Successful in 4s

Details

CI & Build / TypeScript typecheck (push) Successful in 46s

Details

CI & Build / integration (push) Successful in 52s

Details

CI & Build / Python tests (push) Successful in 1m21s

Details

CI & Build / Build & push image (push) Successful in 36s

Details

623464323e

`best_available_score` was added by #3670 so a bar could be judged from
what it rejected, and it arrived null on four unrelated causes: the corpus
offered nothing, the query was empty, the embedder was down, or the
DATABASE QUERY FAILED. Only the first is a measurement. The fourth is the
#2663 shape — a swallowed failure rendering as a clean zero — inside the
field added to fix an instance of the #2663 shape.

Found while trying to explain why reuse_slot returned nothing on 45 of 45
calls, and auto_inject on 153 of 161. That investigation is still open;
what it established first is that the readout could not answer it.

THE FIX IS NOT A NEW COLUMN. A call that never searched writes no row, so
every remaining null means one thing: searched, and nothing came close.
That is the convention the pre-tool arm already follows for a blank
command — "a row here would report a call that never happened and drag the
clear-rate down with phantom declines" — extended from the case a caller
can see in advance to the ones only the search knows about.

Both searches stamp `report["searched"]` FALSE before anything can return
and True only where a real result set exists, so every early return leaves
it false. It has to be the first thing done to the dict: a return added
above that line would leave the key absent.

ABSENT IS A THIRD STATE AND IT DEFAULTS TO TRUE. A caller that passes no
report cannot know, and the safe reading there is the old behaviour. Only
a real search can report False, so absent means "nobody asked" and never
"it failed" — which is also why 66 existing mocked searches across twelve
test files keep working unchanged rather than being rewritten to simulate
a flag they do not care about.

A FAILURE IS NOT MADE INVISIBLE. semantic_search_notes already logs a
WARNING on a query failure, which is where a broken search belongs: a
counter cannot say "I am broken" without a reader already trusting it.

Three tests, and the middle one is what makes them discriminate — a
blanket `return` passes the first and fails the second, because a call
that searched and came back empty is the only evidence a threshold is too
high (#3497).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

bvandeusen merged commit cd910e32a0 into main

2026-09-09 13:36:43 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#147