feat(rules): a surfaced rule gets an outcome, not just a read (#4212)

Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and
that it was PULLED. It could not say what happened next, so a rule that fires
constantly and is always obeyed and a rule that fires constantly and is never
obeyed left byte-identical telemetry. The second is far the more urgent and
was the one the readout could not name — measured on a session where three of
seven misses were caught by the operator and none by the system.

Two new events, `applied` and `departed`, and a `detail` column carrying the
why of a departure. No CHECK migration: `event` was created in 0094 as plain
Text with no constraint, verified in the migration rather than assumed from
the model, so rule 36 does not bite here — said in both places because the
next person adding a value will reach for it.

THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently-
unchanged is the failure this milestone was opened on, and it cannot be
reported: an agent that knew it was ignoring a rule would not be ignoring it.
So nothing here asks. `applied` and `departed` are reported; the third state
is a rule that was opened and left no trace. An `ignored` enum member would
collect nothing while reading as though it had measured something, which is
#3311's failure — a statistic that could not vary being taken for a finding.

`detail` is a column rather than two more bare event strings because a
departure stripped of its reason reads back as a miss, so the two states this
exists to separate would collapse again one layer down, in the readout, where
nobody would see it happen. Nullable: following a rule needs no argument, and
an expensive event is one that stops being recorded.

`outcome_state` is the single reading of the four states, taking the aggregate
`usage_for_rules` already returns, so the badge, the readout and any later
session summary cannot disagree about what "followed" means — the drift #3246
found across the rules system. A departure outranks an application: a rule
both applied and argued with is a rule someone argued with, and the argument
is the half worth surfacing.

`rule_outcome` is the MCP door, classed as a WRITE. The read-only set
tolerates getters that call record_pulled, but those are reads that leave a
trace; this tool's entire effect is the row, and the row carries prose the
agent authored. A read-scoped key that can put text in the operator's
database is not read-scoped, whatever table it lands in.

Backup carries `detail` on both sides. It is the one field here a fresh
install cannot re-earn — counts come back by being used again, a stated
reason exists once — and #4197 records that the column guard watches the
export side only, so the round-trip test is the thing that would catch a
one-sided add.

Delivery is deliberately not settled here: how an agent gets prompted to
record an outcome is step 3's subject, and the same record serves whichever
answer that step reaches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

tests/test_integration_backup_rule_usage_roundtrip.py

@@ -25,7 +25,9 @@ from sqlalchemy import select
 
 from scribe.models import async_session
 from scribe.models.note import Note
-from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
+from scribe.models.rule_usage import (
+    APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
+)
 from scribe.models.rulebook import Rule, Rulebook, RulebookTopic
 from scribe.models.user import User
 from scribe.services import backup
@@ -141,6 +143,21 @@ async def source():
                 user_id=None, rule_id=rule.id,
                 event=SURFACED, source="write_path_rule",
             ),
+            # A departure and its reason (#4212). Seeded here because
+            # `detail` is the ONE field on this table that cannot be
+            # recomputed: a fresh install re-earns its counts by being used,
+            # but a stated reason exists once and is gone if a restore drops
+            # it — and a `departed` row that comes back reasonless reads as a
+            # rule that was simply missed.
+            RuleUsageEvent(
+                user_id=uid, rule_id=rule.id,
+                event=DEPARTED, source="mcp_rule_outcome",
+                detail="the integration lane has no registry credentials",
+            ),
+            RuleUsageEvent(
+                user_id=uid, rule_id=rule.id,
+                event=APPLIED, source="mcp_rule_outcome",
+            ),
         ])
         await s.commit()
         book_id, rule_id, note_id = book.id, rule.id, note.id
@@ -232,8 +249,8 @@ async def restored(source):
 
 async def test_every_event_comes_back(restored):
     """The count first: every shape assertion below reads the same on an empty
-    list, so without this a restore that dropped all three would pass them."""
-    assert len(restored["events"]) == 3
+    list, so without this a restore that dropped all five would pass them."""
+    assert len(restored["events"]) == 5
 
 
 async def test_the_events_attach_to_the_RESTORED_rule(restored):
@@ -272,7 +289,9 @@ async def test_the_actor_is_remapped_and_a_missing_one_survives(restored):
     surfacings of exactly the surface being measured."""
     attributed = [e for e in restored["events"] if e.user_id is not None]
     orphaned = [e for e in restored["events"] if e.user_id is None]
-    assert len(attributed) == 2
+    # Four attributed: the surfacing, the pull, and the two outcome rows
+    # added with `detail` (#4212). One orphaned, deliberately.
+    assert len(attributed) == 4
     assert len(orphaned) == 1, (
         "the event with no actor did not come back. Telemetry outlives the "
         "account it was recorded for; dropping it silently lowers the "
@@ -289,6 +308,40 @@ async def test_the_event_and_source_survive(restored):
     assert pairs == {
         (SURFACED, "write_path_rule"),
         (PULLED, "mcp_get_rule"),
+        (DEPARTED, "mcp_rule_outcome"),
+        (APPLIED, "mcp_rule_outcome"),
     }
     assert sum(1 for e in restored["events"] if e.event == SURFACED) == 2
     assert sum(1 for e in restored["events"] if e.event == PULLED) == 1
+
+
+async def test_a_departures_reason_survives_the_round_trip(restored):
+    """The one field here that a fresh install cannot re-earn.
+
+    Counts come back by being used again; a stated reason exists once. A
+    restore that kept the `departed` row and dropped its `detail` would turn
+    a deliberate, argued departure into something indistinguishable from a
+    rule that was read and missed — which is the exact distinction milestone
+    419 was opened to create, undone silently at the one moment nobody is
+    watching.
+
+    #4197 is the standing warning behind this test: the backup column guard
+    watches the export side only, so a column added to the model and to the
+    exporter and NOT to the importer round-trips as null with nothing to say
+    so.
+    """
+    departures = [e for e in restored["events"] if e.event == DEPARTED]
+    assert len(departures) == 1
+    assert departures[0].detail == (
+        "the integration lane has no registry credentials"
+    )
+
+
+async def test_an_application_carries_no_reason_and_that_is_not_a_loss(restored):
+    """`applied` is the unremarkable case and is stored reasonless on
+    purpose. Asserted so that a later change making `detail` NOT NULL — or
+    backfilling it with a placeholder — has to argue with a test rather than
+    quietly make every application look like it had something to say."""
+    applications = [e for e in restored["events"] if e.event == APPLIED]
+    assert len(applications) == 1
+    assert applications[0].detail is None

tests/test_mcp_tool_rulebooks.py

@@ -478,3 +478,107 @@ async def test_move_rule_on_someone_elses_rule_is_not_found():
         from scribe.mcp.tools.rulebooks import move_rule
         with pytest.raises(ValueError, match="not found"):
             await move_rule(rule_id=94, project_id=3)
+
+
+# ── rule_outcome: what a rule actually changed (#4212, milestone 419) ─────
+
+@pytest.mark.asyncio
+async def test_rule_outcome_records_an_application():
+    rule = fake_rule(id=156, title="`dev` is home")
+    with patch(
+        "scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
+        AsyncMock(return_value=rule),
+    ), patch(
+        "scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
+    ) as rec:
+        from scribe.mcp.tools.rulebooks import rule_outcome
+        out = await rule_outcome(rule_id=156, outcome="applied")
+    assert out["outcome"] == "applied" and out["recorded"] is True
+    assert out["why"] is None
+    assert rec.call_args.kwargs["outcome"] == "applied"
+    assert rec.call_args.kwargs["source"] == "mcp_rule_outcome"
+
+
+@pytest.mark.asyncio
+async def test_rule_outcome_records_a_departure_with_its_reason():
+    rule = fake_rule(id=156, title="`dev` is home")
+    with patch(
+        "scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
+        AsyncMock(return_value=rule),
+    ), patch(
+        "scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
+    ) as rec:
+        from scribe.mcp.tools.rulebooks import rule_outcome
+        out = await rule_outcome(
+            rule_id=156, outcome="departed", why="the operator asked for main"
+        )
+    assert out["outcome"] == "departed"
+    assert rec.call_args.kwargs["detail"] == "the operator asked for main"
+
+
+@pytest.mark.asyncio
+async def test_a_departure_without_a_reason_is_refused_at_the_door():
+    """Refused with a message that says WHY a reason is needed, not just
+    that one is missing — the caller is an agent deciding whether to bother,
+    and "it cannot be told from a miss" is the argument that lands."""
+    rule = fake_rule(id=156, title="`dev` is home")
+    with patch(
+        "scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
+        AsyncMock(return_value=rule),
+    ), patch(
+        "scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
+    ) as rec:
+        from scribe.mcp.tools.rulebooks import rule_outcome
+        with pytest.raises(ValueError, match="departure needs its reason"):
+            await rule_outcome(rule_id=156, outcome="departed", why="   ")
+    rec.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_there_is_no_way_to_report_having_ignored_a_rule():
+    """Deliberate, and the reason is in the tool's docstring: noticing that
+    you ignored a rule is the same act as not ignoring it, so the state is
+    derived rather than reported. A caller reaching for the word gets an
+    error rather than a row that would read as measurement."""
+    rule = fake_rule(id=156, title="`dev` is home")
+    with patch(
+        "scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
+        AsyncMock(return_value=rule),
+    ), patch(
+        "scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
+    ) as rec:
+        from scribe.mcp.tools.rulebooks import rule_outcome
+        for bogus in ("ignored", "skipped", "read", ""):
+            with pytest.raises(ValueError, match="must be 'applied' or 'departed'"):
+                await rule_outcome(rule_id=156, outcome=bogus)
+    rec.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_rule_outcome_refuses_a_rule_the_caller_cannot_see():
+    """The access check comes FIRST, so a miss cannot be used to probe for
+    the existence of someone else's rule, and nothing is recorded against
+    an id the caller has no claim on."""
+    with patch(
+        "scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
+        AsyncMock(return_value=None),
+    ), patch(
+        "scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
+    ) as rec:
+        from scribe.mcp.tools.rulebooks import rule_outcome
+        with pytest.raises(ValueError, match="rule 999 not found"):
+            await rule_outcome(rule_id=999, outcome="applied")
+    rec.assert_not_called()
+
+
+def test_rule_outcome_is_registered_and_is_not_read_only():
+    """Its whole effect is a write, and `why` is prose the agent authored —
+    a read-scoped key that can put text in the operator's database is not
+    read-scoped, whatever table it lands in."""
+    from scribe.mcp.server import _READ_ONLY_TOOLS, _WRITE_TOOLS
+    from scribe.mcp.tools import rulebooks as mod
+    mcp = FakeMCP()
+    mod.register(mcp)
+    assert "rule_outcome" in mcp.names
+    assert "rule_outcome" in _WRITE_TOOLS
+    assert "rule_outcome" not in _READ_ONLY_TOOLS

tests/test_services_rule_usage.py

@@ -6,7 +6,9 @@ where a mistake is silent rather than loud.
 """
 import pytest
 
-from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
+from scribe.models.rule_usage import (
+    APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
+)
 from scribe.services import rule_usage
 
 
@@ -101,6 +103,9 @@ def test_the_zero_readout_names_every_key():
         "surfaced_count": 0,
         "ambient_count": 0,
         "pull_count": 0,
+        "applied_count": 0,
+        "departed_count": 0,
+        "last_outcome_at": None,
         "last_surfaced_at": None,
         "last_pulled_at": None,
     }
@@ -255,3 +260,134 @@ async def test_a_preloaded_rule_does_not_read_as_a_ranked_surfacing(_dispose_eng
                 delete(RuleUsageEvent).where(RuleUsageEvent.user_id == 990021)
             )
             await s.commit()
+
+
+# ── the outcome stream (#4212, milestone 419) ─────────────────────────────
+#
+# What these guard is a distinction, not a payload. Before this existed, a
+# rule read and obeyed and a rule read and ignored left byte-identical
+# telemetry, so the readout could not name the failure the whole milestone
+# was opened on. The tests that matter most below are the ones asserting
+# that a REASONLESS DEPARTURE IS NEVER WRITTEN, and that an unacted rule is
+# a state in its own right rather than the absence of one.
+
+def test_an_applied_outcome_needs_no_argument(captured):
+    """Following a rule is the ordinary case. Charging prose for it would
+    make the cheap event expensive, and an expensive event stops being
+    recorded — which costs the whole measurement."""
+    rule_usage.record_rule_outcome(
+        user_id=7, rule_id=156, outcome=APPLIED, source="mcp_rule_outcome"
+    )
+    [batch] = captured
+    assert batch == [{
+        "user_id": 7, "rule_id": 156, "event": APPLIED,
+        "source": "mcp_rule_outcome", "detail": None,
+    }]
+
+
+def test_a_departure_carries_its_reason(captured):
+    rule_usage.record_rule_outcome(
+        user_id=7, rule_id=156, outcome=DEPARTED, source="mcp_rule_outcome",
+        detail="  the integration lane has no registry credentials  ",
+    )
+    [batch] = captured
+    assert batch[0]["event"] == DEPARTED
+    assert batch[0]["detail"] == "the integration lane has no registry credentials"
+
+
+@pytest.mark.parametrize("reason", ["", "   ", "\n", None])
+def test_a_departure_with_no_reason_is_never_written(captured, reason):
+    """THE ONE THAT MATTERS. A `departed` row without its why reads back as a
+    miss, so writing one would collapse the two states this table exists to
+    separate — silently, in the readout, where nobody would see it happen.
+    Dropped and reported, never stored: telemetry that lies is worse than
+    telemetry that is absent (#2663)."""
+    rule_usage.record_rule_outcome(
+        user_id=7, rule_id=156, outcome=DEPARTED,
+        source="mcp_rule_outcome", detail=reason or "",
+    )
+    assert captured == []
+
+
+def test_an_unknown_outcome_is_never_written(captured):
+    """Including the one somebody will reach for. There is no `ignored`
+    event by design — see `record_rule_outcome` — and a caller inventing one
+    must not get a row that reads as though the state were measurable."""
+    for bogus in ("ignored", "skipped", "surfaced", "", "APPLIED "):
+        rule_usage.record_rule_outcome(
+            user_id=7, rule_id=156, outcome=bogus, source="mcp_rule_outcome"
+        )
+    assert captured == []
+
+
+def test_an_outcome_row_is_one_row(captured):
+    """A judgement is about one rule. Unlike a surfacing, which delivers a
+    whole hint at once, there is no batch shape to get wrong here — asserted
+    so that a later 'helpful' bulk variant has to change a test that says
+    why."""
+    rule_usage.record_rule_outcome(
+        user_id=7, rule_id=1, outcome=APPLIED, source="mcp_rule_outcome"
+    )
+    [batch] = captured
+    assert len(batch) == 1
+
+
+# ── the four states, read off the aggregate ───────────────────────────────
+
+def _usage(**kw):
+    base = rule_usage.empty_rule_usage()
+    base.update(kw)
+    return base
+
+
+def test_a_rule_surfaced_and_never_opened_is_unread():
+    assert rule_usage.outcome_state(_usage(surfaced_count=4)) == rule_usage.UNREAD
+
+
+def test_a_rule_opened_and_acted_on_is_followed():
+    assert rule_usage.outcome_state(
+        _usage(surfaced_count=4, pull_count=1, applied_count=1)
+    ) == rule_usage.FOLLOWED
+
+
+def test_a_rule_opened_and_departed_from_is_departed():
+    assert rule_usage.outcome_state(
+        _usage(surfaced_count=4, pull_count=1, departed_count=1)
+    ) == rule_usage.DEPARTED_FROM
+
+
+def test_a_rule_opened_and_never_acted_on_is_unacted() -> None:
+    """THE STATE THAT DID NOT EXIST, and the reason for the milestone. Not
+    "no data": the rule was surfaced, deliberately opened, and then left no
+    trace of having mattered. Until now that was arithmetically identical to
+    compliance, which is why nothing could report it."""
+    assert rule_usage.outcome_state(
+        _usage(surfaced_count=4, pull_count=2)
+    ) == rule_usage.UNACTED
+
+
+def test_unacted_and_followed_are_not_the_same_reading():
+    """Stated as its own test because it IS the milestone in one line. If a
+    change ever makes these two agree, the measurement is gone and every
+    other test here would still pass."""
+    opened_only = _usage(surfaced_count=4, pull_count=2)
+    opened_and_applied = _usage(surfaced_count=4, pull_count=2, applied_count=1)
+    assert rule_usage.outcome_state(opened_only) != rule_usage.outcome_state(
+        opened_and_applied
+    )
+
+
+def test_a_departure_outranks_an_application():
+    """A rule both applied and argued with is a rule someone argued with, and
+    the argument is the half worth surfacing. Reporting it as plain
+    compliance would bury the one row a reader most wants to see."""
+    assert rule_usage.outcome_state(
+        _usage(pull_count=3, applied_count=5, departed_count=1)
+    ) == rule_usage.DEPARTED_FROM
+
+
+def test_the_state_reads_the_aggregate_the_readout_already_returns():
+    """`outcome_state` takes `usage_for_rules`' own shape, so the badge, the
+    readout and any later session summary cannot disagree about what
+    "followed" means — the drift #3246 found across the rules system."""
+    assert rule_usage.outcome_state(rule_usage.empty_rule_usage()) == rule_usage.UNREAD