feat(rules): a surfaced rule gets an outcome, not just a read (#4212)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped

Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and
that it was PULLED. It could not say what happened next, so a rule that fires
constantly and is always obeyed and a rule that fires constantly and is never
obeyed left byte-identical telemetry. The second is far the more urgent and
was the one the readout could not name — measured on a session where three of
seven misses were caught by the operator and none by the system.

Two new events, `applied` and `departed`, and a `detail` column carrying the
why of a departure. No CHECK migration: `event` was created in 0094 as plain
Text with no constraint, verified in the migration rather than assumed from
the model, so rule 36 does not bite here — said in both places because the
next person adding a value will reach for it.

THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently-
unchanged is the failure this milestone was opened on, and it cannot be
reported: an agent that knew it was ignoring a rule would not be ignoring it.
So nothing here asks. `applied` and `departed` are reported; the third state
is a rule that was opened and left no trace. An `ignored` enum member would
collect nothing while reading as though it had measured something, which is
#3311's failure — a statistic that could not vary being taken for a finding.

`detail` is a column rather than two more bare event strings because a
departure stripped of its reason reads back as a miss, so the two states this
exists to separate would collapse again one layer down, in the readout, where
nobody would see it happen. Nullable: following a rule needs no argument, and
an expensive event is one that stops being recorded.

`outcome_state` is the single reading of the four states, taking the aggregate
`usage_for_rules` already returns, so the badge, the readout and any later
session summary cannot disagree about what "followed" means — the drift #3246
found across the rules system. A departure outranks an application: a rule
both applied and argued with is a rule someone argued with, and the argument
is the half worth surfacing.

`rule_outcome` is the MCP door, classed as a WRITE. The read-only set
tolerates getters that call record_pulled, but those are reads that leave a
trace; this tool's entire effect is the row, and the row carries prose the
agent authored. A read-scoped key that can put text in the operator's
database is not read-scoped, whatever table it lands in.

Backup carries `detail` on both sides. It is the one field here a fresh
install cannot re-earn — counts come back by being used again, a stated
reason exists once — and #4197 records that the column guard watches the
export side only, so the round-trip test is the thing that would catch a
one-sided add.

Delivery is deliberately not settled here: how an agent gets prompted to
record an outcome is step 3's subject, and the same record serves whichever
answer that step reaches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-20 23:56:05 -04:00
co-authored by Claude Opus 5
parent 0fe19a8440
commit dfcb000719
9 changed files with 615 additions and 11 deletions
@@ -25,7 +25,9 @@ from sqlalchemy import select
from scribe.models import async_session
from scribe.models.note import Note
from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
from scribe.models.rule_usage import (
APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
)
from scribe.models.rulebook import Rule, Rulebook, RulebookTopic
from scribe.models.user import User
from scribe.services import backup
@@ -141,6 +143,21 @@ async def source():
user_id=None, rule_id=rule.id,
event=SURFACED, source="write_path_rule",
),
# A departure and its reason (#4212). Seeded here because
# `detail` is the ONE field on this table that cannot be
# recomputed: a fresh install re-earns its counts by being used,
# but a stated reason exists once and is gone if a restore drops
# it — and a `departed` row that comes back reasonless reads as a
# rule that was simply missed.
RuleUsageEvent(
user_id=uid, rule_id=rule.id,
event=DEPARTED, source="mcp_rule_outcome",
detail="the integration lane has no registry credentials",
),
RuleUsageEvent(
user_id=uid, rule_id=rule.id,
event=APPLIED, source="mcp_rule_outcome",
),
])
await s.commit()
book_id, rule_id, note_id = book.id, rule.id, note.id
@@ -232,8 +249,8 @@ async def restored(source):
async def test_every_event_comes_back(restored):
"""The count first: every shape assertion below reads the same on an empty
list, so without this a restore that dropped all three would pass them."""
assert len(restored["events"]) == 3
list, so without this a restore that dropped all five would pass them."""
assert len(restored["events"]) == 5
async def test_the_events_attach_to_the_RESTORED_rule(restored):
@@ -272,7 +289,9 @@ async def test_the_actor_is_remapped_and_a_missing_one_survives(restored):
surfacings of exactly the surface being measured."""
attributed = [e for e in restored["events"] if e.user_id is not None]
orphaned = [e for e in restored["events"] if e.user_id is None]
assert len(attributed) == 2
# Four attributed: the surfacing, the pull, and the two outcome rows
# added with `detail` (#4212). One orphaned, deliberately.
assert len(attributed) == 4
assert len(orphaned) == 1, (
"the event with no actor did not come back. Telemetry outlives the "
"account it was recorded for; dropping it silently lowers the "
@@ -289,6 +308,40 @@ async def test_the_event_and_source_survive(restored):
assert pairs == {
(SURFACED, "write_path_rule"),
(PULLED, "mcp_get_rule"),
(DEPARTED, "mcp_rule_outcome"),
(APPLIED, "mcp_rule_outcome"),
}
assert sum(1 for e in restored["events"] if e.event == SURFACED) == 2
assert sum(1 for e in restored["events"] if e.event == PULLED) == 1
async def test_a_departures_reason_survives_the_round_trip(restored):
"""The one field here that a fresh install cannot re-earn.
Counts come back by being used again; a stated reason exists once. A
restore that kept the `departed` row and dropped its `detail` would turn
a deliberate, argued departure into something indistinguishable from a
rule that was read and missed — which is the exact distinction milestone
419 was opened to create, undone silently at the one moment nobody is
watching.
#4197 is the standing warning behind this test: the backup column guard
watches the export side only, so a column added to the model and to the
exporter and NOT to the importer round-trips as null with nothing to say
so.
"""
departures = [e for e in restored["events"] if e.event == DEPARTED]
assert len(departures) == 1
assert departures[0].detail == (
"the integration lane has no registry credentials"
)
async def test_an_application_carries_no_reason_and_that_is_not_a_loss(restored):
"""`applied` is the unremarkable case and is stored reasonless on
purpose. Asserted so that a later change making `detail` NOT NULL — or
backfilling it with a placeholder — has to argue with a test rather than
quietly make every application look like it had something to say."""
applications = [e for e in restored["events"] if e.event == APPLIED]
assert len(applications) == 1
assert applications[0].detail is None