Files
FabledScribe/tests/test_preference_write_path.py
T
bvandeusenandClaude Opus 5 07bdbf1647
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Failing after 33s
CI & Build / Python tests (push) Failing after 37s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / Build & push image (push) Skipped
feat(rules)!: a rule cannot be created, or edited into, having no trigger (#4099)
`when_to_apply` is not metadata. `rule_document` embeds a rule as
`{title} — {trigger}` / `When to apply: {trigger}\n\n{statement}`, the trigger
appearing twice so purpose dominates a short vector — the shape note 2485
measured on snippets (a 0.153 top-to-second gap against 0.010–0.023 for
everything else). Without one the document silently becomes title + statement:
a DIFFERENT shape, ranked against a corpus it does not match, with nothing to
report it. Every bar and every rank in the system assumes one shape.

`create_preference` has refused an empty trigger since it shipped. The two rule
creators defaulted it to "" — so the shape was enforced for the record kind
that guides and optional for the kind that binds.

The guard lives in the SERVICE, because both doors reach it: the MCP tools and
the frontend's fast path in routes/rulebooks.py. Written in either alone, the
other could still create a rule that never fires. The route keeps a matching
check for the STATUS CODE only (400, not the 404 it maps ValueError to).

update_rule refuses to EMPTY an existing trigger, checked after the mutation so
it covers `clear=[...]`, an emptied form input, and any route added later.
Deliberately asked as "did this edit remove one" rather than "does one exist":
a rule predating the guard has none, and refusing to save it would freeze
precisely the unreachable records that most need fixing.

Deliberately not following arose_from_id, which the human door exempts itself
from because provenance is about auditing what the AGENT changed. That reasoning
does not reach this field — a missing trigger is not a missing explanation, it
is a rule that does not work, and it fails an operator as badly as a session.

15 test fixtures across 6 files were creating rules with no trigger. They now
pass one; that they did not is the point — curation is not a guarantee.

Step 1 of milestone 416 "Retrieval stops guessing a bar". First because every
later step assumes one document shape, and it is much cheaper to guarantee
before a corpus grows than to backfill after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-16 14:45:32 -04:00

227 lines
9.7 KiB
Python

"""The preference write path, and how it differs from a rule's (milestone 399).
WHY TWO DOORS AT ALL
`create_rule`'s docstring IS the approval gate (#3557): it tells its caller to
propose, offer three answers, and wait. That is right for a rule — the person
a rule binds should have agreed to be bound.
A preference inverts it. The operator's framing: *"preferences are rules that
scribe can and should update during use."* A preference that asks every time
never drifts, and drifting is the whole feature. Reaching one through
`create_rule(kind=...)` would mean reading it through the gate's prose, and
when_to_apply="when the moment this fixture stands in for arises",
the caller would hesitate over exactly the act this kind exists to make
routine.
So the asymmetry is the product, and these tests pin it.
TWO PRESENCE CHECKS, NEVER AN ABSENCE
The tempting guard is "create_preference's docstring does NOT run the approval
loop". That is the shape snippet #3352 warns against: an absence check passes
against a docstring that has been deleted, emptied, or rewritten into
something else entirely, and it reads as coverage while proving nothing.
So the asymmetry is asserted as two PRESENCE facts — the rule door still asks,
the preference door still says write it — and each fails if its own side is
tidied away. Synonym families, structure not wording, the same bargain
test_rule_creation_asks_first strikes.
"""
from unittest.mock import AsyncMock, patch
import pytest
from tests.helpers import fake_rule, plain_rule_detail as _plain_detail
from tests.helpers import tool_doc as _doc
# The tool layer reads its caller from a ContextVar the HTTP transport sets.
# With no request in flight, the module binds it itself (snippet #2836).
pytestmark = pytest.mark.usefixtures("_bind_user")
MODULE = "scribe.mcp.tools.rulebooks"
# ── the required fields, and why each is required ───────────────────────
@pytest.mark.asyncio
async def test_a_preference_without_a_trigger_is_refused():
"""A preference with no `when_to_apply` is inert, not merely incomplete.
The trigger is two-thirds of the embedded document, so a record without
one never surfaces at the moment it applies. Refusing at the tool is the
difference between an error the writer can fix and a preference that is
written, stored, and silently never delivered — which looks identical to
one nobody wrote.
"""
create_mock = AsyncMock()
with patch(f"{MODULE}.rulebooks_svc.create_rule", create_mock):
from scribe.mcp.tools.rulebooks import create_preference
with pytest.raises(ValueError, match="when_to_apply is required"):
await create_preference(
topic_id=10, title="t", statement="s",
when_to_apply=" ", arose_from_id=42,
)
create_mock.assert_not_called()
@pytest.mark.asyncio
async def test_a_preference_without_provenance_is_refused():
"""Provenance is the price of the ungated write.
A preference is expected to change as the work teaches it. A corpus that
drifts with no record of what taught each change is one nobody can audit —
and the operator's veto over drift depends entirely on being able to read
why it happened.
"""
create_mock = AsyncMock()
with patch(f"{MODULE}.rulebooks_svc.create_rule", create_mock):
from scribe.mcp.tools.rulebooks import create_preference
with pytest.raises(ValueError, match="arose_from_id is required"):
await create_preference(
topic_id=10, title="t", statement="s",
when_to_apply="when x", arose_from_id=0,
)
create_mock.assert_not_called()
@pytest.mark.asyncio
async def test_create_preference_stores_the_preference_kind():
"""The tool's one irreducible job.
Asserted on the kwarg reaching the service rather than on the returned
payload: a tool that accepted the call and wrote a plain rule would
return something that reads correctly, and the force would be wrong.
"""
rule = fake_rule(id=100, kind="preference")
create_mock = AsyncMock(return_value=rule)
with patch(f"{MODULE}.rulebooks_svc.create_rule", create_mock), _plain_detail():
from scribe.mcp.tools.rulebooks import create_preference
await create_preference(
topic_id=10, title="Pace hard debugging",
statement="One step per turn.",
when_to_apply="during hard debugging",
arose_from_id=42,
)
kwargs = create_mock.call_args.kwargs
assert kwargs["kind"] == "preference"
assert kwargs["arose_from_id"] == 42
assert kwargs["when_to_apply"] == "during hard debugging"
@pytest.mark.asyncio
async def test_a_near_duplicate_preference_blocks():
"""The gate is what lets this corpus be written freely and stay small.
The second preference about a thing must UPDATE the first. Two that
quietly disagree are worse than none: retrieval surfaces whichever scores
higher, and nobody learns the other exists.
"""
from scribe.services.dedup import DuplicateMatch
dup = DuplicateMatch(id=47, title="Pace hard debugging", similarity=1.0, reason="title")
create_mock = AsyncMock()
with patch(f"{MODULE}.dedup_svc.find_duplicate_rule", AsyncMock(return_value=dup)), \
patch(f"{MODULE}.rulebooks_svc.create_rule", create_mock):
from scribe.mcp.tools.rulebooks import create_preference
out = await create_preference(
topic_id=10, title="Pace hard debugging", statement="s",
when_to_apply="when", arose_from_id=42,
)
assert out["duplicate"] is True
assert out["existing_id"] == 47
create_mock.assert_not_called()
@pytest.mark.asyncio
async def test_updating_a_preference_without_provenance_is_refused():
update_mock = AsyncMock()
with patch(f"{MODULE}.rulebooks_svc.update_rule", update_mock):
from scribe.mcp.tools.rulebooks import update_preference
with pytest.raises(ValueError, match="arose_from_id is required"):
await update_preference(rule_id=5, arose_from_id=0, statement="new")
update_mock.assert_not_called()
@pytest.mark.asyncio
async def test_update_preference_forwards_what_taught_the_change():
rule = fake_rule(id=5, kind="preference")
update_mock = AsyncMock(return_value=rule)
with patch(f"{MODULE}.rulebooks_svc.update_rule", update_mock), _plain_detail():
from scribe.mcp.tools.rulebooks import update_preference
await update_preference(
rule_id=5, arose_from_id=99, statement="the new way",
)
kwargs = update_mock.call_args.kwargs
assert kwargs["arose_from_id"] == 99
assert kwargs["statement"] == "the new way"
# ── the asymmetry, as two presence facts ────────────────────────────────
def test_the_rule_door_still_asks_before_writing():
"""Half one of the asymmetry. If this fails, the gate was tidied away and
preferences are no longer the exception — they are just the same thing.
"""
doc = _doc(MODULE, "create_rule").lower()
asks = ("approve", "propose", "ask", "question")
assert any(w in doc for w in asks), (
"create_rule's docstring no longer runs the propose-then-approve loop. "
"The preference path's whole justification is that it is the exception "
"to this; with the gate gone there is no asymmetry left to justify."
)
def test_the_preference_door_says_to_write_it():
"""Half two. The inverting instruction has to be PRESENT, not merely
unaccompanied by a gate.
An agent that has internalised #3557 will hesitate to write or rewrite a
preference unless told plainly that this door is different. Silence here
does not read as permission — it reads as an omission, and the caller
falls back on the behaviour it already knows.
"""
for tool in ("create_preference", "update_preference"):
doc = _doc(MODULE, tool).lower()
permits = ("expected", "no approval", "without asking", "ordinary work",
"write it", "not a liberty", "no proposal")
assert any(w in doc for w in permits), (
f"{tool}'s docstring no longer tells its caller that writing "
"without an approval loop is expected. A caller carrying "
"create_rule's gate will default to asking, and a preference "
"nothing ever updates is a rule nobody enforces."
)
def test_the_preference_door_names_the_force_distinction():
"""The routing test, stated positively (rule 165).
The confusion this milestone exists to fix is that a session cannot tell
which kind it is holding. If the docstring stops drawing the line, the
tool becomes a second way to write rules.
"""
doc = _doc(MODULE, "create_preference").lower()
assert "rule" in doc and any(
w in doc for w in ("followed", "binds", "breaks", "consistency")
), (
"create_preference's docstring no longer distinguishes a preference "
"from a rule by force. Without that line the tool is a second door "
"onto the rulebook with a weaker gate."
)
def test_the_preference_door_keeps_the_record_out_of_scope():
"""Preferences shape HOW work is done, never WHAT is recorded.
Worth pinning because "record it the way I like it" is the natural next
reach, and it would make dev-logs and issues idiosyncratic per author —
while the record is the one thing that has to outlive the person.
"""
doc = _doc(MODULE, "create_preference").lower()
assert "record" in doc, (
"create_preference's docstring no longer says that a preference does "
"not change what gets recorded. That boundary is the one a reader "
"would cross without noticing."
)