feat(rules): a surfaced rule gets an outcome, not just a read (#4212)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m3s
CI & Build / Python tests (push) Failing after 1m6s
CI & Build / Build & push image (push) Skipped
Milestone 419 step 1. `rule_usage_events` could say a rule was SURFACED and that it was PULLED. It could not say what happened next, so a rule that fires constantly and is always obeyed and a rule that fires constantly and is never obeyed left byte-identical telemetry. The second is far the more urgent and was the one the readout could not name — measured on a session where three of seven misses were caught by the operator and none by the system. Two new events, `applied` and `departed`, and a `detail` column carrying the why of a departure. No CHECK migration: `event` was created in 0094 as plain Text with no constraint, verified in the migration rather than assumed from the model, so rule 36 does not bite here — said in both places because the next person adding a value will reach for it. THE THIRD STATE IS DERIVED, AND THAT IS THE DESIGN. Read-and-silently- unchanged is the failure this milestone was opened on, and it cannot be reported: an agent that knew it was ignoring a rule would not be ignoring it. So nothing here asks. `applied` and `departed` are reported; the third state is a rule that was opened and left no trace. An `ignored` enum member would collect nothing while reading as though it had measured something, which is #3311's failure — a statistic that could not vary being taken for a finding. `detail` is a column rather than two more bare event strings because a departure stripped of its reason reads back as a miss, so the two states this exists to separate would collapse again one layer down, in the readout, where nobody would see it happen. Nullable: following a rule needs no argument, and an expensive event is one that stops being recorded. `outcome_state` is the single reading of the four states, taking the aggregate `usage_for_rules` already returns, so the badge, the readout and any later session summary cannot disagree about what "followed" means — the drift #3246 found across the rules system. A departure outranks an application: a rule both applied and argued with is a rule someone argued with, and the argument is the half worth surfacing. `rule_outcome` is the MCP door, classed as a WRITE. The read-only set tolerates getters that call record_pulled, but those are reads that leave a trace; this tool's entire effect is the row, and the row carries prose the agent authored. A read-scoped key that can put text in the operator's database is not read-scoped, whatever table it lands in. Backup carries `detail` on both sides. It is the one field here a fresh install cannot re-earn — counts come back by being used again, a stated reason exists once — and #4197 records that the column guard watches the export side only, so the round-trip test is the thing that would catch a one-sided add. Delivery is deliberately not settled here: how an agent gets prompted to record an outcome is step 3's subject, and the same record serves whichever answer that step reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -25,7 +25,9 @@ from sqlalchemy import select
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.note import Note
|
||||
from scribe.models.rule_usage import PULLED, SURFACED, RuleUsageEvent
|
||||
from scribe.models.rule_usage import (
|
||||
APPLIED, DEPARTED, PULLED, SURFACED, RuleUsageEvent,
|
||||
)
|
||||
from scribe.models.rulebook import Rule, Rulebook, RulebookTopic
|
||||
from scribe.models.user import User
|
||||
from scribe.services import backup
|
||||
@@ -141,6 +143,21 @@ async def source():
|
||||
user_id=None, rule_id=rule.id,
|
||||
event=SURFACED, source="write_path_rule",
|
||||
),
|
||||
# A departure and its reason (#4212). Seeded here because
|
||||
# `detail` is the ONE field on this table that cannot be
|
||||
# recomputed: a fresh install re-earns its counts by being used,
|
||||
# but a stated reason exists once and is gone if a restore drops
|
||||
# it — and a `departed` row that comes back reasonless reads as a
|
||||
# rule that was simply missed.
|
||||
RuleUsageEvent(
|
||||
user_id=uid, rule_id=rule.id,
|
||||
event=DEPARTED, source="mcp_rule_outcome",
|
||||
detail="the integration lane has no registry credentials",
|
||||
),
|
||||
RuleUsageEvent(
|
||||
user_id=uid, rule_id=rule.id,
|
||||
event=APPLIED, source="mcp_rule_outcome",
|
||||
),
|
||||
])
|
||||
await s.commit()
|
||||
book_id, rule_id, note_id = book.id, rule.id, note.id
|
||||
@@ -232,8 +249,8 @@ async def restored(source):
|
||||
|
||||
async def test_every_event_comes_back(restored):
|
||||
"""The count first: every shape assertion below reads the same on an empty
|
||||
list, so without this a restore that dropped all three would pass them."""
|
||||
assert len(restored["events"]) == 3
|
||||
list, so without this a restore that dropped all five would pass them."""
|
||||
assert len(restored["events"]) == 5
|
||||
|
||||
|
||||
async def test_the_events_attach_to_the_RESTORED_rule(restored):
|
||||
@@ -272,7 +289,9 @@ async def test_the_actor_is_remapped_and_a_missing_one_survives(restored):
|
||||
surfacings of exactly the surface being measured."""
|
||||
attributed = [e for e in restored["events"] if e.user_id is not None]
|
||||
orphaned = [e for e in restored["events"] if e.user_id is None]
|
||||
assert len(attributed) == 2
|
||||
# Four attributed: the surfacing, the pull, and the two outcome rows
|
||||
# added with `detail` (#4212). One orphaned, deliberately.
|
||||
assert len(attributed) == 4
|
||||
assert len(orphaned) == 1, (
|
||||
"the event with no actor did not come back. Telemetry outlives the "
|
||||
"account it was recorded for; dropping it silently lowers the "
|
||||
@@ -289,6 +308,40 @@ async def test_the_event_and_source_survive(restored):
|
||||
assert pairs == {
|
||||
(SURFACED, "write_path_rule"),
|
||||
(PULLED, "mcp_get_rule"),
|
||||
(DEPARTED, "mcp_rule_outcome"),
|
||||
(APPLIED, "mcp_rule_outcome"),
|
||||
}
|
||||
assert sum(1 for e in restored["events"] if e.event == SURFACED) == 2
|
||||
assert sum(1 for e in restored["events"] if e.event == PULLED) == 1
|
||||
|
||||
|
||||
async def test_a_departures_reason_survives_the_round_trip(restored):
|
||||
"""The one field here that a fresh install cannot re-earn.
|
||||
|
||||
Counts come back by being used again; a stated reason exists once. A
|
||||
restore that kept the `departed` row and dropped its `detail` would turn
|
||||
a deliberate, argued departure into something indistinguishable from a
|
||||
rule that was read and missed — which is the exact distinction milestone
|
||||
419 was opened to create, undone silently at the one moment nobody is
|
||||
watching.
|
||||
|
||||
#4197 is the standing warning behind this test: the backup column guard
|
||||
watches the export side only, so a column added to the model and to the
|
||||
exporter and NOT to the importer round-trips as null with nothing to say
|
||||
so.
|
||||
"""
|
||||
departures = [e for e in restored["events"] if e.event == DEPARTED]
|
||||
assert len(departures) == 1
|
||||
assert departures[0].detail == (
|
||||
"the integration lane has no registry credentials"
|
||||
)
|
||||
|
||||
|
||||
async def test_an_application_carries_no_reason_and_that_is_not_a_loss(restored):
|
||||
"""`applied` is the unremarkable case and is stored reasonless on
|
||||
purpose. Asserted so that a later change making `detail` NOT NULL — or
|
||||
backfilling it with a placeholder — has to argue with a test rather than
|
||||
quietly make every application look like it had something to say."""
|
||||
applications = [e for e in restored["events"] if e.event == APPLIED]
|
||||
assert len(applications) == 1
|
||||
assert applications[0].detail is None
|
||||
|
||||
Reference in New Issue
Block a user