feat(rules): rules retrieve against the operator's message (#3852)
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 38s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 38s
The third rule arm, and the one the other two cannot reach. `write_path_rule` is keyed on code, `pre_tool_rule` on a command — both things the session is about to DO. A rule that governs what to SAY has no such trigger: extract intent from loose phrasing, raise a conflict before acting, hand off an action with its reason, end a finding with an offer all bind on a RESPONSE, and no tool call precedes one. The operator's message is the only query that exists before a response is composed. That hook searched notes alone, so no rule had ever been retrieved against a thing the operator actually said — and residency was the only surface those rules had, which is what milestone 394 removes. A SEPARATE FUNCTION, not a branch in build_autoinject_hint, because of its early returns. That arm bails when auto-inject is disabled, when the query is blank, when nothing clears the note bar — every one a statement about NOTES. Folded in, an operator who turned the awareness menu off would silently lose their rules, a coupling with no symptom since both look like a quiet hook. Two functions, two sets of gates, composed in the route. Guarded as "the rule arm never asks the notes arm's config", which is the structural fact. Joins _ARMS rather than getting its own test file. #3497's history is that the pre-tool arm inherited a defect from its sibling by being MODELLED on it instead of sharing with it, and a third arm modelled on two is two chances to repeat that. Repeat rendering, fresh-only counting, log-before-bailout, the kind register and the two-recorders identity are properties of every arm or of none. The bar is INHERITED and says so. 0.72 was tuned against code and commands; prose is a different query shape against the same documents, and triggers are written in the vocabulary of the moment — which for most rules is act vocabulary. Starting at the only number with evidence behind it and logging every call from the first deploy is what makes it settleable; guessing lower would put an unmeasured bar in front of a corpus that binds. k=3, anchored on this hook's own budget rather than the act arms'. RULEHINT_LIMIT is 1 because that arm fires before every Bash call; this one fires once per turn, beside a notes menu already spending three slots. And a prompt genuinely contains more than one act — "merge to main and then start on X" is two — where a command is one thing. `prompt_rule` added to RANKED_SOURCES: a ranker picked it, and a ranked source missing from that tuple is silently counted as bulk delivery and drops out of the pull-through denominator. The hook reads and writes the SHARED rule ledger under scribe-priorart, not a private one — one session keeps one list, aged (#3751), so a rule named here is not re-announced before the next Bash call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -444,6 +444,160 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
|
||||
return await pc.build_tool_rule_hint(1, tool, command, **kwargs)
|
||||
|
||||
|
||||
# ── the prompt-boundary arm (#3852) ─────────────────────────────────────
|
||||
#
|
||||
# The third arm, and it joins _ARMS rather than getting a test file of its
|
||||
# own. That is the point of the shared parametrisation: #3497's history is
|
||||
# that the pre-tool arm inherited a defect from its sibling by being MODELLED
|
||||
# on it instead of sharing with it, and a third arm modelled on two is two
|
||||
# chances to repeat that. Everything in the family — repeat rendering,
|
||||
# fresh-only counting, the log-before-bailout order, the kind register, the
|
||||
# two recorders reading one list — is a property of every arm or of none.
|
||||
#
|
||||
# Fewer patches than its siblings because it does less: no prior-art menu, no
|
||||
# config object, no concept query. Just a bar, a search, and two recorders.
|
||||
|
||||
|
||||
def _prompt_patches(pc, hits, recorder, retrieval_log=None):
|
||||
return (
|
||||
# The arm reads its own threshold key rather than a shared config
|
||||
# object — a third corpus with a bar nothing has yet tuned for it.
|
||||
patch.object(pc, "get_setting", AsyncMock(return_value="0.6")),
|
||||
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
|
||||
patch.object(pc, "record_retrieval", retrieval_log or MagicMock()),
|
||||
patch.object(pc, "record_rule_surfaced", recorder),
|
||||
)
|
||||
|
||||
|
||||
async def _run_prompt_arm(hits, recorder, prompt="please merge to main",
|
||||
retrieval_log=None, **kwargs):
|
||||
from scribe.services import plugin_context as pc
|
||||
with ExitStack() as stack:
|
||||
for ctx in _prompt_patches(pc, hits, recorder, retrieval_log=retrieval_log):
|
||||
stack.enter_context(ctx)
|
||||
return await pc.build_prompt_rule_hint(1, prompt, **kwargs)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_prompt_arm_retrieves_against_what_the_operator_SAID():
|
||||
"""The gap this arm closes, stated as the thing that had no trigger.
|
||||
|
||||
Both other arms are keyed on an act — a file write, a command. A rule that
|
||||
governs what to SAY has no act in front of it: extract intent from loose
|
||||
phrasing, raise a conflict before acting, end a finding with an offer all
|
||||
bind on a response. Before this, the operator's message reached only
|
||||
`semantic_search_notes`, so no rule had ever been retrieved against a
|
||||
thing the operator actually said.
|
||||
"""
|
||||
rec = MagicMock()
|
||||
search = AsyncMock(return_value=[(0.79, fake_rule(
|
||||
id=2, title="`main` — never without explicit request",
|
||||
when_to_apply="opening or merging a dev→main pull request",
|
||||
))])
|
||||
from scribe.services import plugin_context as pc
|
||||
with ExitStack() as stack:
|
||||
stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
|
||||
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
|
||||
stack.enter_context(patch.object(pc, "record_retrieval", MagicMock()))
|
||||
stack.enter_context(patch.object(pc, "record_rule_surfaced", rec))
|
||||
out = await pc.build_prompt_rule_hint(1, "please merge to main")
|
||||
|
||||
# The PROMPT is the query — not a path, not a command.
|
||||
assert search.call_args.args[1] == "please merge to main"
|
||||
assert "get_rule(2)" in out["context"]
|
||||
assert rec.call_args.kwargs["source"] == "prompt_rule"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_prompt_arm_addresses_the_request_not_a_tool_call():
|
||||
"""`where` has to name the moment, and this arm's moment is the asking.
|
||||
|
||||
"may apply to this Bash call" would be a lie here — there is no Bash call,
|
||||
which is the entire reason the arm exists.
|
||||
"""
|
||||
out = await _run_prompt_arm(
|
||||
[(0.79, fake_rule(id=77, title="Extract intent from loose phrasing"))],
|
||||
MagicMock(),
|
||||
)
|
||||
assert "may apply to this request" in out["context"], out["context"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_prompt_arm_says_nothing_when_asked_nothing():
|
||||
"""A blank prompt is not a query, and searching on one would put a row in
|
||||
retrieval_logs that no operator action produced."""
|
||||
search = AsyncMock(return_value=[])
|
||||
log = MagicMock()
|
||||
from scribe.services import plugin_context as pc
|
||||
with ExitStack() as stack:
|
||||
stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
|
||||
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
|
||||
stack.enter_context(patch.object(pc, "record_retrieval", log))
|
||||
stack.enter_context(patch.object(pc, "record_rule_surfaced", MagicMock()))
|
||||
out = await pc.build_prompt_rule_hint(1, " ")
|
||||
|
||||
assert out == {"context": "", "rule_ids": []}
|
||||
search.assert_not_called()
|
||||
log.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
|
||||
"""#3497's defect, pinned on the arm that most needs it.
|
||||
|
||||
This bar is INHERITED from the act arms and unverified against prose. The
|
||||
zero rows are therefore the whole evidence base for whether 0.72 belongs
|
||||
here at all — an arm that logged only the calls it liked would report a
|
||||
flawless clear-rate however wrong the number is.
|
||||
"""
|
||||
log = MagicMock()
|
||||
out = await _run_prompt_arm([], MagicMock(), retrieval_log=log)
|
||||
|
||||
assert out["context"] == ""
|
||||
assert log.call_count == 1, (
|
||||
"the prompt arm returned early without logging a call that found "
|
||||
"nothing — the only evidence its inherited threshold is too high"
|
||||
)
|
||||
assert log.call_args.kwargs["source"] == "prompt_rule"
|
||||
assert log.call_args.kwargs["results"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_turning_off_the_notes_menu_does_not_turn_off_rules():
|
||||
"""Why this is a separate function and not a branch in the notes arm.
|
||||
|
||||
`build_autoinject_hint` returns early when auto-inject is disabled, when
|
||||
the query is blank, and when nothing clears the note bar. Every one of
|
||||
those is a statement about NOTES. Folded together, an operator who turned
|
||||
the awareness menu off would silently stop receiving RULES — a coupling
|
||||
with no symptom, since both failure modes look like a quiet hook.
|
||||
|
||||
Pinned as "the rule arm never asks the notes arm's config", which is the
|
||||
structural fact rather than a simulation of the setting. A future refactor
|
||||
that reaches for that config here fails, whatever it then does with it.
|
||||
"""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
notes_cfg = AsyncMock(return_value={
|
||||
"enabled": False, "threshold": 0.55, "top_k": 3,
|
||||
})
|
||||
with ExitStack() as stack:
|
||||
stack.enter_context(patch.object(pc, "get_autoinject_config", notes_cfg))
|
||||
for ctx in _prompt_patches(
|
||||
pc, [(0.79, fake_rule(id=2, title="`main` — never without explicit request"))],
|
||||
MagicMock(),
|
||||
):
|
||||
stack.enter_context(ctx)
|
||||
out = await pc.build_prompt_rule_hint(1, "please merge to main")
|
||||
|
||||
assert "get_rule(2)" in out["context"], (
|
||||
"the rule arm went quiet while the notes menu was disabled — the two "
|
||||
"are different claims with different costs of being missed, and one "
|
||||
"operator setting should not silence both"
|
||||
)
|
||||
notes_cfg.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
|
||||
"""The 2026-09-03 incident in one test: reaching for curl against the forge
|
||||
@@ -871,7 +1025,14 @@ _THREE_HITS = [
|
||||
(0.74, fake_rule(id=161, title="Reach the forge through its MCP tools")),
|
||||
]
|
||||
|
||||
_ARMS = [("write_path_rule", _run_arm), ("pre_tool_rule", _run_tool_arm)]
|
||||
_ARMS = [
|
||||
("write_path_rule", _run_arm),
|
||||
("pre_tool_rule", _run_tool_arm),
|
||||
# The prompt arm joins the family rather than being modelled on it — see
|
||||
# the block above _prompt_patches for why that distinction is the whole
|
||||
# lesson of #3497.
|
||||
("prompt_rule", _run_prompt_arm),
|
||||
]
|
||||
|
||||
|
||||
def _both_ends(log, rec, source):
|
||||
@@ -896,7 +1057,7 @@ def _both_ends(log, rec, source):
|
||||
return logged, surfaced
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.parametrize(
|
||||
("excluded", "expected"),
|
||||
[([], 3), ([157], 2), ([156, 157, 161], 0)],
|
||||
@@ -954,7 +1115,7 @@ _FRESH_TAIL = "not in this session's loaded set"
|
||||
_SEEN_TAIL = "You saw it earlier this session"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
|
||||
"""THE REGRESSION, stated as the thing that used to be absent.
|
||||
@@ -983,7 +1144,7 @@ async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, run):
|
||||
"""One clause differs, and it is the clause that would otherwise be false.
|
||||
@@ -1029,7 +1190,7 @@ _RULE_FORCE = "before deciding it does not apply"
|
||||
_PREF_FORCE = "for how this has been done before"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_preference_does_not_speak_in_the_rules_voice(source, run):
|
||||
"""The register, pinned on the two places force is actually asserted.
|
||||
@@ -1053,7 +1214,7 @@ async def test_a_preference_does_not_speak_in_the_rules_voice(source, run):
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_kind_and_seen_do_not_read_each_other(source, run):
|
||||
"""The structural claim the design rests on: two INDEPENDENT axes.
|
||||
@@ -1093,7 +1254,7 @@ async def test_kind_and_seen_do_not_read_each_other(source, run):
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_neither_tail_injects_the_rule_statement(source, run):
|
||||
"""The budget, pinned on both branches.
|
||||
@@ -1138,7 +1299,7 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
|
||||
# checkable, so it is asserted rather than described.
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool"])
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_reference_is_rendered_but_not_counted(source, run):
|
||||
"""The counters must read exactly as they did before #3750."""
|
||||
|
||||
Reference in New Issue
Block a user