feat(telemetry): retrieval_telemetry says what is wrong (#3431)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 23s

The tool returned distributions and left the reading to the caller, so every
readout was the same four checks done by hand — #3430's baseline, #3835's rule
near-misses, the #1038 rerank gate. Mechanical, and therefore forgettable.

Tonight's acceptance pass on #3898 was the case for doing this. Reading it by
hand meant catching that two surfaces had `covers_window: false`, that
`prompt_rule`'s floor had moved three times inside the window (which made the
readout self-contradictory: deliveries at 0.622 beside refusals at 0.7199),
and that 15 of 20 near-misses were one record against text no operator wrote.
Miss any of those and the obvious conclusion was "the bar is too tight" — a
floor change that would have injected one preference into every notification.

`warnings` is always present and empty when clean, so its emptiness is an
answer rather than a gap. Each entry carries the numbers that produced it:
"345 calls, 0 declined" is the analysis, "check write_path_rule" is an
instruction to redo it. Five codes — cannot_decline, band_hugs_floor,
no_duration, surfaced_never_pulled, unregistered_source.

cannot_decline has three guards, each a bug it would otherwise cause. Asked
surfaces are exempt (a search returning a list every time is working). An arm
not known to log unconditionally is exempt — that is #3497 exactly, where both
rule arms recorded only their hits, so a decline count of zero was a LOGGING
defect and this warning would have sent the reader to a threshold that was
never involved. Unregistered sources get numbers but no verdict.

`silent_surfaces` is the half the rows cannot show: an arm that emitted
nothing is invisible to every row-based check and looks exactly like an arm
that does not exist. It is driven by a new declared registry,
`retrieval_registry.POINTS` — deliberately NOT `retrieval_surfaces.SURFACES`,
which answers "what can be tuned" and excludes the reserved slots because a
budget of 1 is their feature. This answers "what can be measured", and the
reserved slots belong in it precisely because they are judgeable without being
tunable. A test asserts the two cannot drift apart.

The registry test derives sources from the call sites with `ast`, not grep,
and the difference is not theoretical: `wide_net` and `report_preference`
reach their recorder as `source=SOURCE` through a module constant, so a grep
for `source="` is blind to both — the narrowing #3191 warns about. Three sites
pass `source` as a variable and are declared in FAN_OUT_SITES; the test pins
those sites but not the values they can pass, which is why the
`unregistered_source` warning exists to catch the rest at first fire.

Thresholds are settings (rule 25) defaulted so a fresh install with almost no
data produces no warnings at all (rule 115) — a new user's first readout
naming five broken things would be describing the emptiness.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-20 20:24:56 -04:00
co-authored by Claude Opus 5
parent 1f7ff7b215
commit e2c3a5c2b5
5 changed files with 1003 additions and 0 deletions
+47
View File
@@ -396,6 +396,53 @@ It is an UPPER BOUND per surface: a pull records the door it came
they are zeros meaning "could not find out", not "nothing happened" — do
not report a pull-through from a block carrying that flag.
`warnings` IS THE PART TO READ FIRST (#3431). Everything above is a
distribution; this is a verdict, and it exists because the same four
checks were being redone by hand on every reading and were easy to
forget. An EMPTY LIST means checked and clean — it is always present, so
its emptiness is an answer rather than a gap. Each entry carries the
numbers that triggered it, so you can disagree with the rule instead of
having to redo the arithmetic:
- `cannot_decline` — an arm that fires unasked answered every one of its
calls. It cannot say nothing, which means it is not applying a floor.
Only ever raised for unbidden arms known to log unconditionally: a
search returning a list every time is doing its job, and an arm whose
zeros were never written would flag a LOGGING bug while pointing you at
a threshold, which is #3497 exactly.
- `band_hugs_floor` — the weakest tenth of what an arm returns sits on
its floor. The bar is doing the selecting and the score is not, so
moving that floor changes how MUCH you get, not how good it is.
- `no_duration` — rows written without timings. A logging gap, not a slow
arm, and it devalues every other number from that source.
- `surfaced_never_pulled` — distinct records shown and never opened, per
corpus. Read their titles before touching a threshold: a record nobody
opens is usually one whose title does not say when it matters.
- `unregistered_source` — rows under a source missing from
`retrieval_registry`. Its numbers are real; no verdict could be
computed, because nothing says whether it was asked or fired unbidden.
`silent_surfaces` IS THE HALF THE ROWS CANNOT SHOW YOU. Every check above
reads rows, so an arm that produced none is invisible to all of them and
looks exactly like an arm that does not exist. This list is driven by the
declared registry instead: points expected to emit that emitted nothing.
Points that are legitimately quiet — the web-UI-only sources on an install
driven through MCP — are excluded by declaration rather than by silence,
so a justified quiet never reads as a gap. The list stays EMPTY on a
window with little traffic: on a fresh install every point is silent, and
reporting all of them would be describing the emptiness.
WARNINGS ARE COMPUTED OVER THE BLOCKS ABOVE, not over a second query, so
one can never disagree with the numbers printed beside it. A window whose
read failed produces none at all — a verdict over rows that did not load
would describe the outage while appearing to describe the system.
Two thresholds govern them, both settings so an install driven harder can
say so: `retrieval_warn_min_calls` (default 30) is how much traffic a
source needs before its silence means anything, and
`retrieval_warn_floor_epsilon` (default 0.02) is how close to the bar
counts as piled on it.
Scoped to your own telemetry — a retrieval log records what your agent
asked for, query text included, and is not a shared record kind.