feat(telemetry): the readout names rules that were opened and changed nothing (#4213)
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m6s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Successful in 35s
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m6s
CI & Build / Python tests (push) Successful in 1m40s
CI & Build / Build & push image (push) Successful in 35s
Milestone 419 step 2. Step 1 made an outcome recordable; this makes it readable. `retrieval_summary`'s rule block gains `applied`, `departed` and `distinct_rules_acted`, and `_compute_warnings` gains two codes. TWO CODES, NOT ONE WITH A ZERO IN IT. `read_and_unacted` reports rules that were opened and left no outcome, against the ones that did. It only fires once outcomes exist anywhere in the window, because a window with none cannot tell "every rule was ignored" from "nothing calls `rule_outcome` yet" — and on every install the day this ships, the truth is the second. Claiming the first there would be #3311's failure exactly: a statistic that could not vary being read as a fact about the corpus. The cold case gets its own code, `outcomes_never_recorded`, whose prose says in as many words that it does NOT mean the rules were ignored. `applied` AND `departed` ARE NOT SUMMED. A departure carries the reason the agent gave and is evidence about the RULE; an application is evidence about the agent. Folded together they would say only "an outcome exists", which is true of both and useful about neither. `distinct_rules_acted` counts either, because for the unacted arithmetic the distinction does not matter. An outcome is not a pull. The fold branches on OUTCOMES first and never routes an outcome through the surfaced/ambient split: `source` on an outcome row names the door the outcome came through, not a ranker, so the ambient distinction has nothing to say about it. An integration test holds that line — if an outcome leaked into the pull counters the silently-unchanged rule would vanish into a compliant-looking total, which is the confusion #4212 was opened to end. Verified by lifting the shipped `_compute_warnings` out of source with `ast` and exercising it against the six populations the new tests assert: cold instrument, warm instrument, departures-only, full compliance, nothing opened, and a failed read. The integration tests for the new counts run against real Postgres in CI — count(distinct) with an IN over an unconstrained column is a SQL shape a mock would agree with whatever it did, which is what #2663 was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -643,6 +643,72 @@ async def test_ambient_alone_reports_no_ratio(_dispose_engine):
|
||||
await cleanup()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_outcome_counts_come_back_split(_dispose_engine):
|
||||
"""A pull says the rule was read; an outcome says it changed something
|
||||
(#4213). Integration rather than a mock because `distinct_rules_acted` is
|
||||
a count(distinct) with an IN over a column that carries no CHECK — the
|
||||
kind of SQL shape #2663 was, where a mock agrees with whatever the code
|
||||
does including nothing.
|
||||
|
||||
`applied` and `departed` stay APART. Summed they would say "an outcome was
|
||||
recorded", which is true of both and useful about neither: a departure is
|
||||
evidence about the rule, an application is evidence about the agent.
|
||||
"""
|
||||
from scribe.services.retrieval_telemetry import retrieval_summary
|
||||
|
||||
cleanup = await _rule_events(990016, [
|
||||
# Opened and followed.
|
||||
(5301, "surfaced", "write_path_rule"),
|
||||
(5301, "pulled", "mcp_get_rule"),
|
||||
(5301, "applied", "mcp_rule_outcome"),
|
||||
# Opened and deliberately departed from.
|
||||
(5302, "surfaced", "write_path_rule"),
|
||||
(5302, "pulled", "mcp_get_rule"),
|
||||
(5302, "departed", "mcp_rule_outcome"),
|
||||
# Opened, and nothing after it. The state the milestone exists for,
|
||||
# and the one that is counted by its ABSENCE.
|
||||
(5303, "surfaced", "write_path_rule"),
|
||||
(5303, "pulled", "mcp_get_rule"),
|
||||
])
|
||||
try:
|
||||
ru = (await retrieval_summary(990016, days=30))["rule_usage"]
|
||||
assert ru["applied"] == 1
|
||||
assert ru["departed"] == 1
|
||||
assert ru["distinct_rules_acted"] == 2
|
||||
assert ru["distinct_rules_pulled"] == 3
|
||||
|
||||
# An outcome is NOT a pull. If `applied` leaked into the pull counters
|
||||
# the silently-unchanged rule would vanish into a compliant-looking
|
||||
# total, which is the exact confusion #4212 was opened to end.
|
||||
assert ru["pulled"] == 3
|
||||
finally:
|
||||
await cleanup()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_outcome_is_never_split_by_ambient(_dispose_engine):
|
||||
"""`source` on an outcome row names the door the outcome came through, not
|
||||
a ranker, so the surfaced/ambient split has nothing to say about it. An
|
||||
outcome recorded from an unranked source must still count."""
|
||||
from scribe.services.retrieval_telemetry import retrieval_summary
|
||||
|
||||
cleanup = await _rule_events(990017, [
|
||||
(5401, "surfaced", "session_start"),
|
||||
(5401, "pulled", "mcp_get_rule"),
|
||||
(5401, "applied", "some_door_nobody_has_ranked"),
|
||||
])
|
||||
try:
|
||||
ru = (await retrieval_summary(990017, days=30))["rule_usage"]
|
||||
assert ru["applied"] == 1
|
||||
assert ru["distinct_rules_acted"] == 1
|
||||
assert ru["ambient"] == 1, "the surfacing was ambient; the outcome is not"
|
||||
finally:
|
||||
await cleanup()
|
||||
|
||||
|
||||
# ── Window coverage (#3712) ────────────────────────────────────────────
|
||||
#
|
||||
# A counter added last week, read over a 30-day window, reports a real count
|
||||
|
||||
Reference in New Issue
Block a user