feat(telemetry): the readout names rules that were opened and changed nothing (#4213)
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m6s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Successful in 35s
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m6s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Successful in 35s
Milestone 419 step 2. Step 1 made an outcome recordable; this makes it readable. `retrieval_summary`'s rule block gains `applied`, `departed` and `distinct_rules_acted`, and `_compute_warnings` gains two codes. TWO CODES, NOT ONE WITH A ZERO IN IT. `read_and_unacted` reports rules that were opened and left no outcome, against the ones that did. It only fires once outcomes exist anywhere in the window, because a window with none cannot tell "every rule was ignored" from "nothing calls `rule_outcome` yet" — and on every install the day this ships, the truth is the second. Claiming the first there would be #3311's failure exactly: a statistic that could not vary being read as a fact about the corpus. The cold case gets its own code, `outcomes_never_recorded`, whose prose says in as many words that it does NOT mean the rules were ignored. `applied` AND `departed` ARE NOT SUMMED. A departure carries the reason the agent gave and is evidence about the RULE; an application is evidence about the agent. Folded together they would say only "an outcome exists", which is true of both and useful about neither. `distinct_rules_acted` counts either, because for the unacted arithmetic the distinction does not matter. An outcome is not a pull. The fold branches on OUTCOMES first and never routes an outcome through the surfaced/ambient split: `source` on an outcome row names the door the outcome came through, not a ranker, so the ambient distinction has nothing to say about it. An integration test holds that line — if an outcome leaked into the pull counters the silently-unchanged rule would vanish into a compliant-looking total, which is the confusion #4212 was opened to end. Verified by lifting the shipped `_compute_warnings` out of source with `ast` and exercising it against the six populations the new tests assert: cold instrument, warm instrument, departures-only, full compliance, nothing opened, and a failed read. The integration tests for the new counts run against real Postgres in CI — count(distinct) with an IN over an unconstrained column is a SQL shape a mock would agree with whatever it did, which is what #2663 was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -336,6 +336,18 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
`pulled_by_human`, the distinct-rule counts, and `pull_through` on the same
|
||||
definition (agent pulls over RANKED surfacings).
|
||||
|
||||
`applied`, `departed` AND `distinct_rules_acted` ARE WHAT HAPPENED AFTER
|
||||
THE RULE WAS OPENED (#4213). A pull says the rule was read; these say it
|
||||
changed something. `applied` counts rules followed, `departed` rules
|
||||
deliberately not followed — kept apart rather than summed, because a
|
||||
departure carries the reason the agent gave and is evidence about the
|
||||
RULE, while an application is evidence about the agent. There is
|
||||
deliberately no count of rules read and quietly ignored: that state is what
|
||||
is left over when a rule was pulled and neither outcome arrived, and
|
||||
`read_and_unacted` below is where it is reported. Asking an agent to
|
||||
declare it would be asking it to notice an omission it is defined by not
|
||||
noticing.
|
||||
|
||||
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
|
||||
with that is the mistake to avoid. The corpora differ by orders of
|
||||
magnitude — a few dozen eligible rules against thousands of notes — so a
|
||||
@@ -418,6 +430,21 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
- `surfaced_never_pulled` — distinct records shown and never opened, per
|
||||
corpus. Read their titles before touching a threshold: a record nobody
|
||||
opens is usually one whose title does not say when it matters.
|
||||
- `read_and_unacted` — distinct rules OPENED in the window that recorded
|
||||
no outcome, against the ones that did. The failure milestone 419 was
|
||||
opened on, and the worse sibling of `surfaced_never_pulled` above: a
|
||||
rule nobody opens is cheap, while a rule read and silently unchanged is
|
||||
indistinguishable from one that worked. It does not say which of the two
|
||||
causes it is — a rule mis-triggering, arriving where it does not apply,
|
||||
or a rule being ignored — and those want opposite fixes, so read the
|
||||
rules before moving anything.
|
||||
- `outcomes_never_recorded` — rules were opened and NOT ONE outcome exists
|
||||
anywhere in the window. Deliberately a separate code, and not a
|
||||
`read_and_unacted` with a zero in it: a window with no outcomes at all
|
||||
cannot tell "every rule was ignored" from "nothing on this install calls
|
||||
`rule_outcome` yet", and reporting the first would manufacture a finding
|
||||
out of an unwired feature. Wire the outcome call before reading this as
|
||||
a fact about the corpus.
|
||||
- `unregistered_source` — rows under a source missing from
|
||||
`retrieval_registry`. Its numbers are real; no verdict could be
|
||||
computed, because nothing says whether it was asked or fired unbidden.
|
||||
|
||||
Reference in New Issue
Block a user