feat(rules): rules retrieve against the operator's message (#3852)
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 38s

The third rule arm, and the one the other two cannot reach. `write_path_rule`
is keyed on code, `pre_tool_rule` on a command — both things the session is
about to DO. A rule that governs what to SAY has no such trigger: extract
intent from loose phrasing, raise a conflict before acting, hand off an
action with its reason, end a finding with an offer all bind on a RESPONSE,
and no tool call precedes one.

The operator's message is the only query that exists before a response is
composed. That hook searched notes alone, so no rule had ever been retrieved
against a thing the operator actually said — and residency was the only
surface those rules had, which is what milestone 394 removes.

A SEPARATE FUNCTION, not a branch in build_autoinject_hint, because of its
early returns. That arm bails when auto-inject is disabled, when the query is
blank, when nothing clears the note bar — every one a statement about NOTES.
Folded in, an operator who turned the awareness menu off would silently lose
their rules, a coupling with no symptom since both look like a quiet hook.
Two functions, two sets of gates, composed in the route. Guarded as "the rule
arm never asks the notes arm's config", which is the structural fact.

Joins _ARMS rather than getting its own test file. #3497's history is that
the pre-tool arm inherited a defect from its sibling by being MODELLED on it
instead of sharing with it, and a third arm modelled on two is two chances to
repeat that. Repeat rendering, fresh-only counting, log-before-bailout, the
kind register and the two-recorders identity are properties of every arm or
of none.

The bar is INHERITED and says so. 0.72 was tuned against code and commands;
prose is a different query shape against the same documents, and triggers are
written in the vocabulary of the moment — which for most rules is act
vocabulary. Starting at the only number with evidence behind it and logging
every call from the first deploy is what makes it settleable; guessing lower
would put an unmeasured bar in front of a corpus that binds.

k=3, anchored on this hook's own budget rather than the act arms'.
RULEHINT_LIMIT is 1 because that arm fires before every Bash call; this one
fires once per turn, beside a notes menu already spending three slots. And a
prompt genuinely contains more than one act — "merge to main and then start
on X" is two — where a command is one thing.

`prompt_rule` added to RANKED_SOURCES: a ranker picked it, and a ranked
source missing from that tuple is silently counted as bulk delivery and drops
out of the pull-through denominator.

The hook reads and writes the SHARED rule ledger under scribe-priorart, not a
private one — one session keeps one list, aged (#3751), so a rule named here
is not re-announced before the next Bash call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-11 07:56:19 -04:00
co-authored by Claude Opus 5
parent 8406871085
commit 44e0b0541f
6 changed files with 376 additions and 10 deletions
+169 -8
View File
@@ -444,6 +444,160 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
return await pc.build_tool_rule_hint(1, tool, command, **kwargs)
# ── the prompt-boundary arm (#3852) ─────────────────────────────────────
#
# The third arm, and it joins _ARMS rather than getting a test file of its
# own. That is the point of the shared parametrisation: #3497's history is
# that the pre-tool arm inherited a defect from its sibling by being MODELLED
# on it instead of sharing with it, and a third arm modelled on two is two
# chances to repeat that. Everything in the family — repeat rendering,
# fresh-only counting, the log-before-bailout order, the kind register, the
# two recorders reading one list — is a property of every arm or of none.
#
# Fewer patches than its siblings because it does less: no prior-art menu, no
# config object, no concept query. Just a bar, a search, and two recorders.
def _prompt_patches(pc, hits, recorder, retrieval_log=None):
return (
# The arm reads its own threshold key rather than a shared config
# object — a third corpus with a bar nothing has yet tuned for it.
patch.object(pc, "get_setting", AsyncMock(return_value="0.6")),
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
patch.object(pc, "record_retrieval", retrieval_log or MagicMock()),
patch.object(pc, "record_rule_surfaced", recorder),
)
async def _run_prompt_arm(hits, recorder, prompt="please merge to main",
retrieval_log=None, **kwargs):
from scribe.services import plugin_context as pc
with ExitStack() as stack:
for ctx in _prompt_patches(pc, hits, recorder, retrieval_log=retrieval_log):
stack.enter_context(ctx)
return await pc.build_prompt_rule_hint(1, prompt, **kwargs)
@pytest.mark.asyncio
async def test_the_prompt_arm_retrieves_against_what_the_operator_SAID():
"""The gap this arm closes, stated as the thing that had no trigger.
Both other arms are keyed on an act — a file write, a command. A rule that
governs what to SAY has no act in front of it: extract intent from loose
phrasing, raise a conflict before acting, end a finding with an offer all
bind on a response. Before this, the operator's message reached only
`semantic_search_notes`, so no rule had ever been retrieved against a
thing the operator actually said.
"""
rec = MagicMock()
search = AsyncMock(return_value=[(0.79, fake_rule(
id=2, title="`main` — never without explicit request",
when_to_apply="opening or merging a dev→main pull request",
))])
from scribe.services import plugin_context as pc
with ExitStack() as stack:
stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
stack.enter_context(patch.object(pc, "record_retrieval", MagicMock()))
stack.enter_context(patch.object(pc, "record_rule_surfaced", rec))
out = await pc.build_prompt_rule_hint(1, "please merge to main")
# The PROMPT is the query — not a path, not a command.
assert search.call_args.args[1] == "please merge to main"
assert "get_rule(2)" in out["context"]
assert rec.call_args.kwargs["source"] == "prompt_rule"
@pytest.mark.asyncio
async def test_the_prompt_arm_addresses_the_request_not_a_tool_call():
"""`where` has to name the moment, and this arm's moment is the asking.
"may apply to this Bash call" would be a lie here — there is no Bash call,
which is the entire reason the arm exists.
"""
out = await _run_prompt_arm(
[(0.79, fake_rule(id=77, title="Extract intent from loose phrasing"))],
MagicMock(),
)
assert "may apply to this request" in out["context"], out["context"]
@pytest.mark.asyncio
async def test_the_prompt_arm_says_nothing_when_asked_nothing():
"""A blank prompt is not a query, and searching on one would put a row in
retrieval_logs that no operator action produced."""
search = AsyncMock(return_value=[])
log = MagicMock()
from scribe.services import plugin_context as pc
with ExitStack() as stack:
stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
stack.enter_context(patch.object(pc, "record_retrieval", log))
stack.enter_context(patch.object(pc, "record_rule_surfaced", MagicMock()))
out = await pc.build_prompt_rule_hint(1, " ")
assert out == {"context": "", "rule_ids": []}
search.assert_not_called()
log.assert_not_called()
@pytest.mark.asyncio
async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
"""#3497's defect, pinned on the arm that most needs it.
This bar is INHERITED from the act arms and unverified against prose. The
zero rows are therefore the whole evidence base for whether 0.72 belongs
here at all — an arm that logged only the calls it liked would report a
flawless clear-rate however wrong the number is.
"""
log = MagicMock()
out = await _run_prompt_arm([], MagicMock(), retrieval_log=log)
assert out["context"] == ""
assert log.call_count == 1, (
"the prompt arm returned early without logging a call that found "
"nothing — the only evidence its inherited threshold is too high"
)
assert log.call_args.kwargs["source"] == "prompt_rule"
assert log.call_args.kwargs["results"] == []
@pytest.mark.asyncio
async def test_turning_off_the_notes_menu_does_not_turn_off_rules():
"""Why this is a separate function and not a branch in the notes arm.
`build_autoinject_hint` returns early when auto-inject is disabled, when
the query is blank, and when nothing clears the note bar. Every one of
those is a statement about NOTES. Folded together, an operator who turned
the awareness menu off would silently stop receiving RULES — a coupling
with no symptom, since both failure modes look like a quiet hook.
Pinned as "the rule arm never asks the notes arm's config", which is the
structural fact rather than a simulation of the setting. A future refactor
that reaches for that config here fails, whatever it then does with it.
"""
from scribe.services import plugin_context as pc
notes_cfg = AsyncMock(return_value={
"enabled": False, "threshold": 0.55, "top_k": 3,
})
with ExitStack() as stack:
stack.enter_context(patch.object(pc, "get_autoinject_config", notes_cfg))
for ctx in _prompt_patches(
pc, [(0.79, fake_rule(id=2, title="`main` — never without explicit request"))],
MagicMock(),
):
stack.enter_context(ctx)
out = await pc.build_prompt_rule_hint(1, "please merge to main")
assert "get_rule(2)" in out["context"], (
"the rule arm went quiet while the notes menu was disabled — the two "
"are different claims with different costs of being missed, and one "
"operator setting should not silence both"
)
notes_cfg.assert_not_called()
@pytest.mark.asyncio
async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
"""The 2026-09-03 incident in one test: reaching for curl against the forge
@@ -871,7 +1025,14 @@ _THREE_HITS = [
(0.74, fake_rule(id=161, title="Reach the forge through its MCP tools")),
]
_ARMS = [("write_path_rule", _run_arm), ("pre_tool_rule", _run_tool_arm)]
_ARMS = [
("write_path_rule", _run_arm),
("pre_tool_rule", _run_tool_arm),
# The prompt arm joins the family rather than being modelled on it — see
# the block above _prompt_patches for why that distinction is the whole
# lesson of #3497.
("prompt_rule", _run_prompt_arm),
]
def _both_ends(log, rec, source):
@@ -896,7 +1057,7 @@ def _both_ends(log, rec, source):
return logged, surfaced
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.parametrize(
("excluded", "expected"),
[([], 3), ([157], 2), ([156, 157, 161], 0)],
@@ -954,7 +1115,7 @@ _FRESH_TAIL = "not in this session's loaded set"
_SEEN_TAIL = "You saw it earlier this session"
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
"""THE REGRESSION, stated as the thing that used to be absent.
@@ -983,7 +1144,7 @@ async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
)
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, run):
"""One clause differs, and it is the clause that would otherwise be false.
@@ -1029,7 +1190,7 @@ _RULE_FORCE = "before deciding it does not apply"
_PREF_FORCE = "for how this has been done before"
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_a_preference_does_not_speak_in_the_rules_voice(source, run):
"""The register, pinned on the two places force is actually asserted.
@@ -1053,7 +1214,7 @@ async def test_a_preference_does_not_speak_in_the_rules_voice(source, run):
)
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_kind_and_seen_do_not_read_each_other(source, run):
"""The structural claim the design rests on: two INDEPENDENT axes.
@@ -1093,7 +1254,7 @@ async def test_kind_and_seen_do_not_read_each_other(source, run):
)
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_neither_tail_injects_the_rule_statement(source, run):
"""The budget, pinned on both branches.
@@ -1138,7 +1299,7 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
# checkable, so it is asserted rather than described.
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_a_reference_is_rendered_but_not_counted(source, run):
"""The counters must read exactly as they did before #3750."""