feat(telemetry): retrieval_telemetry says what is wrong (#3431)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 23s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 23s
The tool returned distributions and left the reading to the caller, so every readout was the same four checks done by hand — #3430's baseline, #3835's rule near-misses, the #1038 rerank gate. Mechanical, and therefore forgettable. Tonight's acceptance pass on #3898 was the case for doing this. Reading it by hand meant catching that two surfaces had `covers_window: false`, that `prompt_rule`'s floor had moved three times inside the window (which made the readout self-contradictory: deliveries at 0.622 beside refusals at 0.7199), and that 15 of 20 near-misses were one record against text no operator wrote. Miss any of those and the obvious conclusion was "the bar is too tight" — a floor change that would have injected one preference into every notification. `warnings` is always present and empty when clean, so its emptiness is an answer rather than a gap. Each entry carries the numbers that produced it: "345 calls, 0 declined" is the analysis, "check write_path_rule" is an instruction to redo it. Five codes — cannot_decline, band_hugs_floor, no_duration, surfaced_never_pulled, unregistered_source. cannot_decline has three guards, each a bug it would otherwise cause. Asked surfaces are exempt (a search returning a list every time is working). An arm not known to log unconditionally is exempt — that is #3497 exactly, where both rule arms recorded only their hits, so a decline count of zero was a LOGGING defect and this warning would have sent the reader to a threshold that was never involved. Unregistered sources get numbers but no verdict. `silent_surfaces` is the half the rows cannot show: an arm that emitted nothing is invisible to every row-based check and looks exactly like an arm that does not exist. It is driven by a new declared registry, `retrieval_registry.POINTS` — deliberately NOT `retrieval_surfaces.SURFACES`, which answers "what can be tuned" and excludes the reserved slots because a budget of 1 is their feature. This answers "what can be measured", and the reserved slots belong in it precisely because they are judgeable without being tunable. A test asserts the two cannot drift apart. The registry test derives sources from the call sites with `ast`, not grep, and the difference is not theoretical: `wide_net` and `report_preference` reach their recorder as `source=SOURCE` through a module constant, so a grep for `source="` is blind to both — the narrowing #3191 warns about. Three sites pass `source` as a variable and are declared in FAN_OUT_SITES; the test pins those sites but not the values they can pass, which is why the `unregistered_source` warning exists to catch the rest at first fire. Thresholds are settings (rule 25) defaulted so a fresh install with almost no data produces no warnings at all (rule 115) — a new user's first readout naming five broken things would be describing the emptiness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -396,6 +396,53 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
they are zeros meaning "could not find out", not "nothing happened" — do
|
||||
not report a pull-through from a block carrying that flag.
|
||||
|
||||
`warnings` IS THE PART TO READ FIRST (#3431). Everything above is a
|
||||
distribution; this is a verdict, and it exists because the same four
|
||||
checks were being redone by hand on every reading and were easy to
|
||||
forget. An EMPTY LIST means checked and clean — it is always present, so
|
||||
its emptiness is an answer rather than a gap. Each entry carries the
|
||||
numbers that triggered it, so you can disagree with the rule instead of
|
||||
having to redo the arithmetic:
|
||||
|
||||
- `cannot_decline` — an arm that fires unasked answered every one of its
|
||||
calls. It cannot say nothing, which means it is not applying a floor.
|
||||
Only ever raised for unbidden arms known to log unconditionally: a
|
||||
search returning a list every time is doing its job, and an arm whose
|
||||
zeros were never written would flag a LOGGING bug while pointing you at
|
||||
a threshold, which is #3497 exactly.
|
||||
- `band_hugs_floor` — the weakest tenth of what an arm returns sits on
|
||||
its floor. The bar is doing the selecting and the score is not, so
|
||||
moving that floor changes how MUCH you get, not how good it is.
|
||||
- `no_duration` — rows written without timings. A logging gap, not a slow
|
||||
arm, and it devalues every other number from that source.
|
||||
- `surfaced_never_pulled` — distinct records shown and never opened, per
|
||||
corpus. Read their titles before touching a threshold: a record nobody
|
||||
opens is usually one whose title does not say when it matters.
|
||||
- `unregistered_source` — rows under a source missing from
|
||||
`retrieval_registry`. Its numbers are real; no verdict could be
|
||||
computed, because nothing says whether it was asked or fired unbidden.
|
||||
|
||||
`silent_surfaces` IS THE HALF THE ROWS CANNOT SHOW YOU. Every check above
|
||||
reads rows, so an arm that produced none is invisible to all of them and
|
||||
looks exactly like an arm that does not exist. This list is driven by the
|
||||
declared registry instead: points expected to emit that emitted nothing.
|
||||
Points that are legitimately quiet — the web-UI-only sources on an install
|
||||
driven through MCP — are excluded by declaration rather than by silence,
|
||||
so a justified quiet never reads as a gap. The list stays EMPTY on a
|
||||
window with little traffic: on a fresh install every point is silent, and
|
||||
reporting all of them would be describing the emptiness.
|
||||
|
||||
WARNINGS ARE COMPUTED OVER THE BLOCKS ABOVE, not over a second query, so
|
||||
one can never disagree with the numbers printed beside it. A window whose
|
||||
read failed produces none at all — a verdict over rows that did not load
|
||||
would describe the outage while appearing to describe the system.
|
||||
|
||||
Two thresholds govern them, both settings so an install driven harder can
|
||||
say so: `retrieval_warn_min_calls` (default 30) is how much traffic a
|
||||
source needs before its silence means anything, and
|
||||
`retrieval_warn_floor_epsilon` (default 0.02) is how close to the bar
|
||||
counts as piled on it.
|
||||
|
||||
Scoped to your own telemetry — a retrieval log records what your agent
|
||||
asked for, query text included, and is not a shared record kind.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user