feat(telemetry): the readout names rules that were opened and changed nothing (#4213)
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m6s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Successful in 35s

Milestone 419 step 2. Step 1 made an outcome recordable; this makes it
readable. `retrieval_summary`'s rule block gains `applied`, `departed` and
`distinct_rules_acted`, and `_compute_warnings` gains two codes.

TWO CODES, NOT ONE WITH A ZERO IN IT. `read_and_unacted` reports rules that
were opened and left no outcome, against the ones that did. It only fires once
outcomes exist anywhere in the window, because a window with none cannot tell
"every rule was ignored" from "nothing calls `rule_outcome` yet" — and on
every install the day this ships, the truth is the second. Claiming the first
there would be #3311's failure exactly: a statistic that could not vary being
read as a fact about the corpus. The cold case gets its own code,
`outcomes_never_recorded`, whose prose says in as many words that it does NOT
mean the rules were ignored.

`applied` AND `departed` ARE NOT SUMMED. A departure carries the reason the
agent gave and is evidence about the RULE; an application is evidence about
the agent. Folded together they would say only "an outcome exists", which is
true of both and useful about neither. `distinct_rules_acted` counts either,
because for the unacted arithmetic the distinction does not matter.

An outcome is not a pull. The fold branches on OUTCOMES first and never routes
an outcome through the surfaced/ambient split: `source` on an outcome row
names the door the outcome came through, not a ranker, so the ambient
distinction has nothing to say about it. An integration test holds that line —
if an outcome leaked into the pull counters the silently-unchanged rule would
vanish into a compliant-looking total, which is the confusion #4212 was opened
to end.

Verified by lifting the shipped `_compute_warnings` out of source with `ast`
and exercising it against the six populations the new tests assert: cold
instrument, warm instrument, departures-only, full compliance, nothing opened,
and a failed read. The integration tests for the new counts run against real
Postgres in CI — count(distinct) with an IN over an unconstrained column is a
SQL shape a mock would agree with whatever it did, which is what #2663 was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-21 00:13:45 -04:00
co-authored by Claude Opus 5
parent 4ebf478575
commit bb8013928f
4 changed files with 257 additions and 0 deletions
+27
View File
@@ -336,6 +336,18 @@ It is an UPPER BOUND per surface: a pull records the door it came
`pulled_by_human`, the distinct-rule counts, and `pull_through` on the same
definition (agent pulls over RANKED surfacings).
`applied`, `departed` AND `distinct_rules_acted` ARE WHAT HAPPENED AFTER
THE RULE WAS OPENED (#4213). A pull says the rule was read; these say it
changed something. `applied` counts rules followed, `departed` rules
deliberately not followed — kept apart rather than summed, because a
departure carries the reason the agent gave and is evidence about the
RULE, while an application is evidence about the agent. There is
deliberately no count of rules read and quietly ignored: that state is what
is left over when a rule was pulled and neither outcome arrived, and
`read_and_unacted` below is where it is reported. Asking an agent to
declare it would be asking it to notice an omission it is defined by not
noticing.
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
with that is the mistake to avoid. The corpora differ by orders of
magnitude — a few dozen eligible rules against thousands of notes — so a
@@ -418,6 +430,21 @@ It is an UPPER BOUND per surface: a pull records the door it came
- `surfaced_never_pulled` — distinct records shown and never opened, per
corpus. Read their titles before touching a threshold: a record nobody
opens is usually one whose title does not say when it matters.
- `read_and_unacted` — distinct rules OPENED in the window that recorded
no outcome, against the ones that did. The failure milestone 419 was
opened on, and the worse sibling of `surfaced_never_pulled` above: a
rule nobody opens is cheap, while a rule read and silently unchanged is
indistinguishable from one that worked. It does not say which of the two
causes it is — a rule mis-triggering, arriving where it does not apply,
or a rule being ignored — and those want opposite fixes, so read the
rules before moving anything.
- `outcomes_never_recorded` — rules were opened and NOT ONE outcome exists
anywhere in the window. Deliberately a separate code, and not a
`read_and_unacted` with a zero in it: a window with no outcomes at all
cannot tell "every rule was ignored" from "nothing on this install calls
`rule_outcome` yet", and reporting the first would manufacture a finding
out of an unwired feature. Wire the outcome call before reading this as
a fact about the corpus.
- `unregistered_source` — rows under a source missing from
`retrieval_registry`. Its numbers are real; no verdict could be
computed, because nothing says whether it was asked or fired unbidden.