feat(rules): the update surfaces teach the trigger shape, not just the create ones (#3855)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 47s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m26s
CI & Build / Build & push image (push) Successful in 24s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 47s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m26s
CI & Build / Build & push image (push) Successful in 24s
A trigger is two-thirds of a rule's embedded document, so one naming a
CATEGORY rather than a moment collapses the record toward its title and it
never arrives. #3835 measured that across 113 rules; #3855 hit it again on
the eight preferences, where six named a category and two did not.
The split was not carelessness, it was an uneven contract. create_rule has
carried the full argument since c61925b (2026-08-27) and the two
preferences authored that day got good triggers; the six written weeks
earlier got categories. The guidance worked wherever it existed — and it
existed on three of five write surfaces. Both update_* tools were silent,
and the update path is where every RETROFITTED trigger is written, which is
most of them: a trigger that already reads fine as English is the one
nobody rewrites.
So:
- update_rule gains the retrofit case, which is a different trap from the
create case. There the field is empty and the instruction is "write one".
Here one exists, reads perfectly well, and the honest-looking verdict is
that it is fine.
- update_preference gains it too, plus why the field is load-bearing there
specifically: preferences get a reserved slot filled by a kind-filtered
query at limit=1, so the corpus ranks against ITSELF and the trigger is
nearly all that separates one from the next.
- create_preference and create_project_rule now SHOW a moment instead of
describing one. Advice about being concrete that is not itself concrete
is the shape that was already on file while the corpus filled up.
The guard pins one property: a tool taking when_to_apply mentions it. That
is exactly what update_preference failed. The surface list is derived from
register() rather than hand-kept, so a write tool added later is in scope
the day it lands.
Two stronger predicates were written for the softer regression — guidance
kept but abstracted — and both were discarded after falsification: counting
quoted phrases measured ambient quotation and passed the broken version,
and scoping that count to a window failed create_preference while correct.
Rule 167 settles it; the discarded attempts are recorded in the test
docstring so the next author does not repeat them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -0,0 +1,116 @@
|
||||
"""Every rule-write surface documents the trigger it can write (#3855).
|
||||
|
||||
WHY THIS EXISTS
|
||||
|
||||
`when_to_apply` is not documentation. `rule_document()` uses it twice — as
|
||||
the embedded title's second half and again above the body — so it dominates
|
||||
the vector, and a rule whose trigger names a CATEGORY rather than a moment
|
||||
collapses toward its title and never arrives. Measured in #3835 across 113
|
||||
rules, then again in #3855 on eight preferences.
|
||||
|
||||
The corpus damage traced to an uneven contract rather than to careless
|
||||
authors. `create_rule` had carried the full argument since 2026-08-27, and
|
||||
the two preferences written that day got good triggers; the six written
|
||||
before it — "during hard debugging", "when reading any request from the
|
||||
operator" — got categories. The guidance worked wherever it existed. It
|
||||
simply did not exist on either `update_*` surface, which is where every
|
||||
RETROFITTED trigger is written, and retrofitting is most of the work: a
|
||||
trigger that already reads fine as English is the one nobody rewrites.
|
||||
|
||||
WHAT THIS PINS
|
||||
|
||||
One structural property, no wording: a tool that accepts `when_to_apply`
|
||||
mentions it in its docstring. That is exactly what `update_preference`
|
||||
failed before #3855 — the parameter existed and the docstring never said so,
|
||||
which is how six triggers got written against no contract at all.
|
||||
|
||||
THE SURFACE LIST IS DERIVED, NEVER HAND-KEPT. It comes from what
|
||||
`rulebooks.register()` actually hands the server, filtered to signatures
|
||||
taking `when_to_apply`, so a rule-write tool added later is in scope on the
|
||||
day it is added. A hand-kept list has to be remembered by the person least
|
||||
likely to know it exists — the argument `RANKED_SOURCES` makes in
|
||||
services/rule_usage.py, and the reason #3855 happened at all.
|
||||
|
||||
WHAT THIS DELIBERATELY DOES NOT PIN, AND WHY IT IS NOT HERE
|
||||
|
||||
The regression worth catching is the one above's *softer* twin: guidance
|
||||
kept but abstracted back to "name the moment in session vocabulary", which
|
||||
is advice about being concrete that is not itself concrete — the shape that
|
||||
was already on file while the corpus filled with categories.
|
||||
|
||||
Two predicates for it were written and both were discarded, because each was
|
||||
falsified and each failed:
|
||||
|
||||
- Counting quoted multi-word phrases anywhere in the docstring measured
|
||||
ambient quotation, not demonstrated triggers. It passed the broken
|
||||
version by scoring unrelated prose ("what happens if someone doesn't do
|
||||
this"), and prose with an odd number of quote characters produced
|
||||
matches spanning the gap BETWEEN two unrelated phrases.
|
||||
- Scoping that count to a window after each "trigger"/"when_to_apply"
|
||||
mention then failed `create_preference` in its CORRECT state, because
|
||||
its examples sit further from the first mention than any defensible
|
||||
window reaches.
|
||||
|
||||
Rule 167's standard settles it: a check that passes on the broken code reads
|
||||
as coverage and stops anyone looking again, so no check is the better of the
|
||||
two. Whether a docstring teaches the shape WELL stays a reading judgment,
|
||||
and the record of that decision lives here so the next author does not spend
|
||||
the same two rounds discovering it.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.helpers import tool_doc as _doc
|
||||
|
||||
_MODULE = "scribe.mcp.tools.rulebooks"
|
||||
|
||||
|
||||
def _registered_tools() -> list:
|
||||
"""Every function `rulebooks.register()` actually hands to the server.
|
||||
|
||||
Collected by handing `register` a stand-in that records what it is given,
|
||||
rather than by reading the source or repeating the tuple here. The point
|
||||
is that this cannot drift from what ships.
|
||||
"""
|
||||
from scribe.mcp.tools import rulebooks
|
||||
|
||||
collected: list = []
|
||||
|
||||
class _Collector:
|
||||
def tool(self, name=None):
|
||||
def register_one(fn):
|
||||
collected.append(fn)
|
||||
return fn
|
||||
return register_one
|
||||
|
||||
rulebooks.register(_Collector())
|
||||
return collected
|
||||
|
||||
|
||||
def _trigger_writers() -> list[str]:
|
||||
"""The registered tools that can write a rule's trigger."""
|
||||
names = [
|
||||
fn.__name__
|
||||
for fn in _registered_tools()
|
||||
if "when_to_apply" in inspect.signature(fn).parameters
|
||||
]
|
||||
# An empty list would make every case below vacuous and the guard would
|
||||
# pass by having nothing to check — absence read as non-existence, the
|
||||
# #3720 shape. Fail loudly instead.
|
||||
assert names, "no rule-write surface takes when_to_apply; the guard is blind"
|
||||
return sorted(names)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", _trigger_writers())
|
||||
def test_a_trigger_writing_surface_documents_the_field(name):
|
||||
"""A tool that can write a trigger must say what the field is for."""
|
||||
doc = _doc(_MODULE, name)
|
||||
assert "when_to_apply" in doc, (
|
||||
f"{name} accepts when_to_apply but never mentions it in its "
|
||||
"docstring. The tool docstring is the agent-facing contract "
|
||||
"(rule 119), and a trigger written against no contract is the #3855 "
|
||||
"defect: six preferences named a category instead of a moment, and "
|
||||
"a category is not a thing any session ever types. Say what the "
|
||||
"field is for, and show a moment — see create_rule."
|
||||
)
|
||||
Reference in New Issue
Block a user