feat(rules): a surfaced rule gets an outcome, not just a read (#4212)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped

Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and
that it was PULLED. It could not say what happened next, so a rule that fires
constantly and is always obeyed and a rule that fires constantly and is never
obeyed left byte-identical telemetry. The second is far the more urgent and
was the one the readout could not name — measured on a session where three of
seven misses were caught by the operator and none by the system.

Two new events, `applied` and `departed`, and a `detail` column carrying the
why of a departure. No CHECK migration: `event` was created in 0094 as plain
Text with no constraint, verified in the migration rather than assumed from
the model, so rule 36 does not bite here — said in both places because the
next person adding a value will reach for it.

THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently-
unchanged is the failure this milestone was opened on, and it cannot be
reported: an agent that knew it was ignoring a rule would not be ignoring it.
So nothing here asks. `applied` and `departed` are reported; the third state
is a rule that was opened and left no trace. An `ignored` enum member would
collect nothing while reading as though it had measured something, which is
#3311's failure — a statistic that could not vary being taken for a finding.

`detail` is a column rather than two more bare event strings because a
departure stripped of its reason reads back as a miss, so the two states this
exists to separate would collapse again one layer down, in the readout, where
nobody would see it happen. Nullable: following a rule needs no argument, and
an expensive event is one that stops being recorded.

`outcome_state` is the single reading of the four states, taking the aggregate
`usage_for_rules` already returns, so the badge, the readout and any later
session summary cannot disagree about what "followed" means — the drift #3246
found across the rules system. A departure outranks an application: a rule
both applied and argued with is a rule someone argued with, and the argument
is the half worth surfacing.

`rule_outcome` is the MCP door, classed as a WRITE. The read-only set
tolerates getters that call record_pulled, but those are reads that leave a
trace; this tool's entire effect is the row, and the row carries prose the
agent authored. A read-scoped key that can put text in the operator's
database is not read-scoped, whatever table it lands in.

Backup carries `detail` on both sides. It is the one field here a fresh
install cannot re-earn — counts come back by being used again, a stated
reason exists once — and #4197 records that the column guard watches the
export side only, so the round-trip test is the thing that would catch a
one-sided add.

Delivery is deliberately not settled here: how an agent gets prompted to
record an outcome is step 3's subject, and the same record serves whichever
answer that step reaches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-20 23:56:05 -04:00
co-authored by Claude Opus 5
parent 0fe19a8440
commit dfcb000719
9 changed files with 615 additions and 11 deletions
+137 -1
View File
@@ -6,7 +6,9 @@ where a mistake is silent rather than loud.
"""
import pytest
from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
from scribe.models.rule_usage import (
APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
)
from scribe.services import rule_usage
@@ -101,6 +103,9 @@ def test_the_zero_readout_names_every_key():
"surfaced_count": 0,
"ambient_count": 0,
"pull_count": 0,
"applied_count": 0,
"departed_count": 0,
"last_outcome_at": None,
"last_surfaced_at": None,
"last_pulled_at": None,
}
@@ -255,3 +260,134 @@ async def test_a_preloaded_rule_does_not_read_as_a_ranked_surfacing(_dispose_eng
delete(RuleUsageEvent).where(RuleUsageEvent.user_id == 990021)
)
await s.commit()
# ── the outcome stream (#4212, milestone 419) ─────────────────────────────
#
# What these guard is a distinction, not a payload. Before this existed, a
# rule read and obeyed and a rule read and ignored left byte-identical
# telemetry, so the readout could not name the failure the whole milestone
# was opened on. The tests that matter most below are the ones asserting
# that a REASONLESS DEPARTURE IS NEVER WRITTEN, and that an unacted rule is
# a state in its own right rather than the absence of one.
def test_an_applied_outcome_needs_no_argument(captured):
"""Following a rule is the ordinary case. Charging prose for it would
make the cheap event expensive, and an expensive event stops being
recorded — which costs the whole measurement."""
rule_usage.record_rule_outcome(
user_id=7, rule_id=156, outcome=APPLIED, source="mcp_rule_outcome"
)
[batch] = captured
assert batch == [{
"user_id": 7, "rule_id": 156, "event": APPLIED,
"source": "mcp_rule_outcome", "detail": None,
}]
def test_a_departure_carries_its_reason(captured):
rule_usage.record_rule_outcome(
user_id=7, rule_id=156, outcome=DEPARTED, source="mcp_rule_outcome",
detail=" the integration lane has no registry credentials ",
)
[batch] = captured
assert batch[0]["event"] == DEPARTED
assert batch[0]["detail"] == "the integration lane has no registry credentials"
@pytest.mark.parametrize("reason", ["", " ", "\n", None])
def test_a_departure_with_no_reason_is_never_written(captured, reason):
"""THE ONE THAT MATTERS. A `departed` row without its why reads back as a
miss, so writing one would collapse the two states this table exists to
separate — silently, in the readout, where nobody would see it happen.
Dropped and reported, never stored: telemetry that lies is worse than
telemetry that is absent (#2663)."""
rule_usage.record_rule_outcome(
user_id=7, rule_id=156, outcome=DEPARTED,
source="mcp_rule_outcome", detail=reason or "",
)
assert captured == []
def test_an_unknown_outcome_is_never_written(captured):
"""Including the one somebody will reach for. There is no `ignored`
event by design — see `record_rule_outcome` — and a caller inventing one
must not get a row that reads as though the state were measurable."""
for bogus in ("ignored", "skipped", "surfaced", "", "APPLIED "):
rule_usage.record_rule_outcome(
user_id=7, rule_id=156, outcome=bogus, source="mcp_rule_outcome"
)
assert captured == []
def test_an_outcome_row_is_one_row(captured):
"""A judgement is about one rule. Unlike a surfacing, which delivers a
whole hint at once, there is no batch shape to get wrong here — asserted
so that a later 'helpful' bulk variant has to change a test that says
why."""
rule_usage.record_rule_outcome(
user_id=7, rule_id=1, outcome=APPLIED, source="mcp_rule_outcome"
)
[batch] = captured
assert len(batch) == 1
# ── the four states, read off the aggregate ───────────────────────────────
def _usage(**kw):
base = rule_usage.empty_rule_usage()
base.update(kw)
return base
def test_a_rule_surfaced_and_never_opened_is_unread():
assert rule_usage.outcome_state(_usage(surfaced_count=4)) == rule_usage.UNREAD
def test_a_rule_opened_and_acted_on_is_followed():
assert rule_usage.outcome_state(
_usage(surfaced_count=4, pull_count=1, applied_count=1)
) == rule_usage.FOLLOWED
def test_a_rule_opened_and_departed_from_is_departed():
assert rule_usage.outcome_state(
_usage(surfaced_count=4, pull_count=1, departed_count=1)
) == rule_usage.DEPARTED_FROM
def test_a_rule_opened_and_never_acted_on_is_unacted() -> None:
"""THE STATE THAT DID NOT EXIST, and the reason for the milestone. Not
"no data": the rule was surfaced, deliberately opened, and then left no
trace of having mattered. Until now that was arithmetically identical to
compliance, which is why nothing could report it."""
assert rule_usage.outcome_state(
_usage(surfaced_count=4, pull_count=2)
) == rule_usage.UNACTED
def test_unacted_and_followed_are_not_the_same_reading():
"""Stated as its own test because it IS the milestone in one line. If a
change ever makes these two agree, the measurement is gone and every
other test here would still pass."""
opened_only = _usage(surfaced_count=4, pull_count=2)
opened_and_applied = _usage(surfaced_count=4, pull_count=2, applied_count=1)
assert rule_usage.outcome_state(opened_only) != rule_usage.outcome_state(
opened_and_applied
)
def test_a_departure_outranks_an_application():
"""A rule both applied and argued with is a rule someone argued with, and
the argument is the half worth surfacing. Reporting it as plain
compliance would bury the one row a reader most wants to see."""
assert rule_usage.outcome_state(
_usage(pull_count=3, applied_count=5, departed_count=1)
) == rule_usage.DEPARTED_FROM
def test_the_state_reads_the_aggregate_the_readout_already_returns():
"""`outcome_state` takes `usage_for_rules`' own shape, so the badge, the
readout and any later session summary cannot disagree about what
"followed" means — the drift #3246 found across the rules system."""
assert rule_usage.outcome_state(rule_usage.empty_rule_usage()) == rule_usage.UNREAD