feat(rules): a surfaced rule gets an outcome, not just a read (#4212)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and that it was PULLED. It could not say what happened next, so a rule that fires constantly and is always obeyed and a rule that fires constantly and is never obeyed left byte-identical telemetry. The second is far the more urgent and was the one the readout could not name — measured on a session where three of seven misses were caught by the operator and none by the system. Two new events, `applied` and `departed`, and a `detail` column carrying the why of a departure. No CHECK migration: `event` was created in 0094 as plain Text with no constraint, verified in the migration rather than assumed from the model, so rule 36 does not bite here — said in both places because the next person adding a value will reach for it. THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently- unchanged is the failure this milestone was opened on, and it cannot be reported: an agent that knew it was ignoring a rule would not be ignoring it. So nothing here asks. `applied` and `departed` are reported; the third state is a rule that was opened and left no trace. An `ignored` enum member would collect nothing while reading as though it had measured something, which is #3311's failure — a statistic that could not vary being taken for a finding. `detail` is a column rather than two more bare event strings because a departure stripped of its reason reads back as a miss, so the two states this exists to separate would collapse again one layer down, in the readout, where nobody would see it happen. Nullable: following a rule needs no argument, and an expensive event is one that stops being recorded. `outcome_state` is the single reading of the four states, taking the aggregate `usage_for_rules` already returns, so the badge, the readout and any later session summary cannot disagree about what "followed" means — the drift #3246 found across the rules system. A departure outranks an application: a rule both applied and argued with is a rule someone argued with, and the argument is the half worth surfacing. `rule_outcome` is the MCP door, classed as a WRITE. The read-only set tolerates getters that call record_pulled, but those are reads that leave a trace; this tool's entire effect is the row, and the row carries prose the agent authored. A read-scoped key that can put text in the operator's database is not read-scoped, whatever table it lands in. Backup carries `detail` on both sides. It is the one field here a fresh install cannot re-earn — counts come back by being used again, a stated reason exists once — and #4197 records that the column guard watches the export side only, so the round-trip test is the thing that would catch a one-sided add. Delivery is deliberately not settled here: how an agent gets prompted to record an outcome is step 3's subject, and the same record serves whichever answer that step reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -6,7 +6,9 @@ where a mistake is silent rather than loud.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
|
||||
from scribe.models.rule_usage import (
|
||||
APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
|
||||
)
|
||||
from scribe.services import rule_usage
|
||||
|
||||
|
||||
@@ -101,6 +103,9 @@ def test_the_zero_readout_names_every_key():
|
||||
"surfaced_count": 0,
|
||||
"ambient_count": 0,
|
||||
"pull_count": 0,
|
||||
"applied_count": 0,
|
||||
"departed_count": 0,
|
||||
"last_outcome_at": None,
|
||||
"last_surfaced_at": None,
|
||||
"last_pulled_at": None,
|
||||
}
|
||||
@@ -255,3 +260,134 @@ async def test_a_preloaded_rule_does_not_read_as_a_ranked_surfacing(_dispose_eng
|
||||
delete(RuleUsageEvent).where(RuleUsageEvent.user_id == 990021)
|
||||
)
|
||||
await s.commit()
|
||||
|
||||
|
||||
# ── the outcome stream (#4212, milestone 419) ─────────────────────────────
|
||||
#
|
||||
# What these guard is a distinction, not a payload. Before this existed, a
|
||||
# rule read and obeyed and a rule read and ignored left byte-identical
|
||||
# telemetry, so the readout could not name the failure the whole milestone
|
||||
# was opened on. The tests that matter most below are the ones asserting
|
||||
# that a REASONLESS DEPARTURE IS NEVER WRITTEN, and that an unacted rule is
|
||||
# a state in its own right rather than the absence of one.
|
||||
|
||||
def test_an_applied_outcome_needs_no_argument(captured):
|
||||
"""Following a rule is the ordinary case. Charging prose for it would
|
||||
make the cheap event expensive, and an expensive event stops being
|
||||
recorded — which costs the whole measurement."""
|
||||
rule_usage.record_rule_outcome(
|
||||
user_id=7, rule_id=156, outcome=APPLIED, source="mcp_rule_outcome"
|
||||
)
|
||||
[batch] = captured
|
||||
assert batch == [{
|
||||
"user_id": 7, "rule_id": 156, "event": APPLIED,
|
||||
"source": "mcp_rule_outcome", "detail": None,
|
||||
}]
|
||||
|
||||
|
||||
def test_a_departure_carries_its_reason(captured):
|
||||
rule_usage.record_rule_outcome(
|
||||
user_id=7, rule_id=156, outcome=DEPARTED, source="mcp_rule_outcome",
|
||||
detail=" the integration lane has no registry credentials ",
|
||||
)
|
||||
[batch] = captured
|
||||
assert batch[0]["event"] == DEPARTED
|
||||
assert batch[0]["detail"] == "the integration lane has no registry credentials"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reason", ["", " ", "\n", None])
|
||||
def test_a_departure_with_no_reason_is_never_written(captured, reason):
|
||||
"""THE ONE THAT MATTERS. A `departed` row without its why reads back as a
|
||||
miss, so writing one would collapse the two states this table exists to
|
||||
separate — silently, in the readout, where nobody would see it happen.
|
||||
Dropped and reported, never stored: telemetry that lies is worse than
|
||||
telemetry that is absent (#2663)."""
|
||||
rule_usage.record_rule_outcome(
|
||||
user_id=7, rule_id=156, outcome=DEPARTED,
|
||||
source="mcp_rule_outcome", detail=reason or "",
|
||||
)
|
||||
assert captured == []
|
||||
|
||||
|
||||
def test_an_unknown_outcome_is_never_written(captured):
|
||||
"""Including the one somebody will reach for. There is no `ignored`
|
||||
event by design — see `record_rule_outcome` — and a caller inventing one
|
||||
must not get a row that reads as though the state were measurable."""
|
||||
for bogus in ("ignored", "skipped", "surfaced", "", "APPLIED "):
|
||||
rule_usage.record_rule_outcome(
|
||||
user_id=7, rule_id=156, outcome=bogus, source="mcp_rule_outcome"
|
||||
)
|
||||
assert captured == []
|
||||
|
||||
|
||||
def test_an_outcome_row_is_one_row(captured):
|
||||
"""A judgement is about one rule. Unlike a surfacing, which delivers a
|
||||
whole hint at once, there is no batch shape to get wrong here — asserted
|
||||
so that a later 'helpful' bulk variant has to change a test that says
|
||||
why."""
|
||||
rule_usage.record_rule_outcome(
|
||||
user_id=7, rule_id=1, outcome=APPLIED, source="mcp_rule_outcome"
|
||||
)
|
||||
[batch] = captured
|
||||
assert len(batch) == 1
|
||||
|
||||
|
||||
# ── the four states, read off the aggregate ───────────────────────────────
|
||||
|
||||
def _usage(**kw):
|
||||
base = rule_usage.empty_rule_usage()
|
||||
base.update(kw)
|
||||
return base
|
||||
|
||||
|
||||
def test_a_rule_surfaced_and_never_opened_is_unread():
|
||||
assert rule_usage.outcome_state(_usage(surfaced_count=4)) == rule_usage.UNREAD
|
||||
|
||||
|
||||
def test_a_rule_opened_and_acted_on_is_followed():
|
||||
assert rule_usage.outcome_state(
|
||||
_usage(surfaced_count=4, pull_count=1, applied_count=1)
|
||||
) == rule_usage.FOLLOWED
|
||||
|
||||
|
||||
def test_a_rule_opened_and_departed_from_is_departed():
|
||||
assert rule_usage.outcome_state(
|
||||
_usage(surfaced_count=4, pull_count=1, departed_count=1)
|
||||
) == rule_usage.DEPARTED_FROM
|
||||
|
||||
|
||||
def test_a_rule_opened_and_never_acted_on_is_unacted() -> None:
|
||||
"""THE STATE THAT DID NOT EXIST, and the reason for the milestone. Not
|
||||
"no data": the rule was surfaced, deliberately opened, and then left no
|
||||
trace of having mattered. Until now that was arithmetically identical to
|
||||
compliance, which is why nothing could report it."""
|
||||
assert rule_usage.outcome_state(
|
||||
_usage(surfaced_count=4, pull_count=2)
|
||||
) == rule_usage.UNACTED
|
||||
|
||||
|
||||
def test_unacted_and_followed_are_not_the_same_reading():
|
||||
"""Stated as its own test because it IS the milestone in one line. If a
|
||||
change ever makes these two agree, the measurement is gone and every
|
||||
other test here would still pass."""
|
||||
opened_only = _usage(surfaced_count=4, pull_count=2)
|
||||
opened_and_applied = _usage(surfaced_count=4, pull_count=2, applied_count=1)
|
||||
assert rule_usage.outcome_state(opened_only) != rule_usage.outcome_state(
|
||||
opened_and_applied
|
||||
)
|
||||
|
||||
|
||||
def test_a_departure_outranks_an_application():
|
||||
"""A rule both applied and argued with is a rule someone argued with, and
|
||||
the argument is the half worth surfacing. Reporting it as plain
|
||||
compliance would bury the one row a reader most wants to see."""
|
||||
assert rule_usage.outcome_state(
|
||||
_usage(pull_count=3, applied_count=5, departed_count=1)
|
||||
) == rule_usage.DEPARTED_FROM
|
||||
|
||||
|
||||
def test_the_state_reads_the_aggregate_the_readout_already_returns():
|
||||
"""`outcome_state` takes `usage_for_rules`' own shape, so the badge, the
|
||||
readout and any later session summary cannot disagree about what
|
||||
"followed" means — the drift #3246 found across the rules system."""
|
||||
assert rule_usage.outcome_state(rule_usage.empty_rule_usage()) == rule_usage.UNREAD
|
||||
|
||||
Reference in New Issue
Block a user