feat(rules): a surfaced rule gets an outcome, not just a read (#4212)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and that it was PULLED. It could not say what happened next, so a rule that fires constantly and is always obeyed and a rule that fires constantly and is never obeyed left byte-identical telemetry. The second is far the more urgent and was the one the readout could not name — measured on a session where three of seven misses were caught by the operator and none by the system. Two new events, `applied` and `departed`, and a `detail` column carrying the why of a departure. No CHECK migration: `event` was created in 0094 as plain Text with no constraint, verified in the migration rather than assumed from the model, so rule 36 does not bite here — said in both places because the next person adding a value will reach for it. THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently- unchanged is the failure this milestone was opened on, and it cannot be reported: an agent that knew it was ignoring a rule would not be ignoring it. So nothing here asks. `applied` and `departed` are reported; the third state is a rule that was opened and left no trace. An `ignored` enum member would collect nothing while reading as though it had measured something, which is #3311's failure — a statistic that could not vary being taken for a finding. `detail` is a column rather than two more bare event strings because a departure stripped of its reason reads back as a miss, so the two states this exists to separate would collapse again one layer down, in the readout, where nobody would see it happen. Nullable: following a rule needs no argument, and an expensive event is one that stops being recorded. `outcome_state` is the single reading of the four states, taking the aggregate `usage_for_rules` already returns, so the badge, the readout and any later session summary cannot disagree about what "followed" means — the drift #3246 found across the rules system. A departure outranks an application: a rule both applied and argued with is a rule someone argued with, and the argument is the half worth surfacing. `rule_outcome` is the MCP door, classed as a WRITE. The read-only set tolerates getters that call record_pulled, but those are reads that leave a trace; this tool's entire effect is the row, and the row carries prose the agent authored. A read-scoped key that can put text in the operator's database is not read-scoped, whatever table it lands in. Backup carries `detail` on both sides. It is the one field here a fresh install cannot re-earn — counts come back by being used again, a stated reason exists once — and #4197 records that the column guard watches the export side only, so the round-trip test is the thing that would catch a one-sided add. Delivery is deliberately not settled here: how an agent gets prompted to record an outcome is step 3's subject, and the same record serves whichever answer that step reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -478,3 +478,107 @@ async def test_move_rule_on_someone_elses_rule_is_not_found():
|
||||
from scribe.mcp.tools.rulebooks import move_rule
|
||||
with pytest.raises(ValueError, match="not found"):
|
||||
await move_rule(rule_id=94, project_id=3)
|
||||
|
||||
|
||||
# ── rule_outcome: what a rule actually changed (#4212, milestone 419) ─────
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_outcome_records_an_application():
|
||||
rule = fake_rule(id=156, title="`dev` is home")
|
||||
with patch(
|
||||
"scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
|
||||
AsyncMock(return_value=rule),
|
||||
), patch(
|
||||
"scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
|
||||
) as rec:
|
||||
from scribe.mcp.tools.rulebooks import rule_outcome
|
||||
out = await rule_outcome(rule_id=156, outcome="applied")
|
||||
assert out["outcome"] == "applied" and out["recorded"] is True
|
||||
assert out["why"] is None
|
||||
assert rec.call_args.kwargs["outcome"] == "applied"
|
||||
assert rec.call_args.kwargs["source"] == "mcp_rule_outcome"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_outcome_records_a_departure_with_its_reason():
|
||||
rule = fake_rule(id=156, title="`dev` is home")
|
||||
with patch(
|
||||
"scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
|
||||
AsyncMock(return_value=rule),
|
||||
), patch(
|
||||
"scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
|
||||
) as rec:
|
||||
from scribe.mcp.tools.rulebooks import rule_outcome
|
||||
out = await rule_outcome(
|
||||
rule_id=156, outcome="departed", why="the operator asked for main"
|
||||
)
|
||||
assert out["outcome"] == "departed"
|
||||
assert rec.call_args.kwargs["detail"] == "the operator asked for main"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_departure_without_a_reason_is_refused_at_the_door():
|
||||
"""Refused with a message that says WHY a reason is needed, not just
|
||||
that one is missing — the caller is an agent deciding whether to bother,
|
||||
and "it cannot be told from a miss" is the argument that lands."""
|
||||
rule = fake_rule(id=156, title="`dev` is home")
|
||||
with patch(
|
||||
"scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
|
||||
AsyncMock(return_value=rule),
|
||||
), patch(
|
||||
"scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
|
||||
) as rec:
|
||||
from scribe.mcp.tools.rulebooks import rule_outcome
|
||||
with pytest.raises(ValueError, match="departure needs its reason"):
|
||||
await rule_outcome(rule_id=156, outcome="departed", why=" ")
|
||||
rec.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_there_is_no_way_to_report_having_ignored_a_rule():
|
||||
"""Deliberate, and the reason is in the tool's docstring: noticing that
|
||||
you ignored a rule is the same act as not ignoring it, so the state is
|
||||
derived rather than reported. A caller reaching for the word gets an
|
||||
error rather than a row that would read as measurement."""
|
||||
rule = fake_rule(id=156, title="`dev` is home")
|
||||
with patch(
|
||||
"scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
|
||||
AsyncMock(return_value=rule),
|
||||
), patch(
|
||||
"scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
|
||||
) as rec:
|
||||
from scribe.mcp.tools.rulebooks import rule_outcome
|
||||
for bogus in ("ignored", "skipped", "read", ""):
|
||||
with pytest.raises(ValueError, match="must be 'applied' or 'departed'"):
|
||||
await rule_outcome(rule_id=156, outcome=bogus)
|
||||
rec.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rule_outcome_refuses_a_rule_the_caller_cannot_see():
|
||||
"""The access check comes FIRST, so a miss cannot be used to probe for
|
||||
the existence of someone else's rule, and nothing is recorded against
|
||||
an id the caller has no claim on."""
|
||||
with patch(
|
||||
"scribe.mcp.tools.rulebooks.rulebooks_svc.get_rule",
|
||||
AsyncMock(return_value=None),
|
||||
), patch(
|
||||
"scribe.mcp.tools.rulebooks.record_rule_outcome", MagicMock()
|
||||
) as rec:
|
||||
from scribe.mcp.tools.rulebooks import rule_outcome
|
||||
with pytest.raises(ValueError, match="rule 999 not found"):
|
||||
await rule_outcome(rule_id=999, outcome="applied")
|
||||
rec.assert_not_called()
|
||||
|
||||
|
||||
def test_rule_outcome_is_registered_and_is_not_read_only():
|
||||
"""Its whole effect is a write, and `why` is prose the agent authored —
|
||||
a read-scoped key that can put text in the operator's database is not
|
||||
read-scoped, whatever table it lands in."""
|
||||
from scribe.mcp.server import _READ_ONLY_TOOLS, _WRITE_TOOLS
|
||||
from scribe.mcp.tools import rulebooks as mod
|
||||
mcp = FakeMCP()
|
||||
mod.register(mcp)
|
||||
assert "rule_outcome" in mcp.names
|
||||
assert "rule_outcome" in _WRITE_TOOLS
|
||||
assert "rule_outcome" not in _READ_ONLY_TOOLS
|
||||
|
||||
Reference in New Issue
Block a user