Retire the always-on tier — every rule arrives by retrieval (milestone 394) #152

Merged
bvandeusen merged 13 commits from dev into main 2026-09-11 16:43:20 -04:00
2 changed files with 106 additions and 46 deletions
Showing only changes of commit 8e06cdf749 - Show all commits
+33 -13
View File
@@ -420,6 +420,14 @@ async def create_rule(
brought it back as the top hit. Where a rule prevents a specific
failure, put that failure's vocabulary here — the error text,
the wrong behaviour, the dead end.
The two spellings, side by side:
RETRIEVES: "the migration failed with a check violation on a
column we just extended"
COLLAPSES: "when working on migrations"
The second names a CATEGORY. No session ever produces a
category — it produces the command, the error, the half-formed
ask — so a trigger written that way leaves the embedded
document to be carried by the title alone.
tier: "always_on" (default) or "conditional".
The test: can you name the trigger WITHOUT naming a system, an
artifact type or a moment? If the honest answer is "whenever you
@@ -519,10 +527,11 @@ async def create_project_rule(
when_to_apply: WHEN this rule fires — the trigger, not the
instruction, and the rule's retrieval surface: name the SYMPTOM,
the words someone would type while stuck. Show the moment rather
than classifying it"the migration failed with a check
violation on a column we just extended" retrieves; "when working
on migrations" does not, because it names a category no session
ever types. See create_rule for the full argument. It informs the
than classifying it:
RETRIEVES: "the CI job passed locally and fails on the runner
with a permission error"
COLLAPSES: "when touching CI config"
See create_rule for the full argument. It informs the
tier below rather than deciding it,
since a project rule's tier turns on area-scope, not on whether
the trigger can be named.
@@ -619,7 +628,12 @@ async def update_rule(
written, the operator's actual message — and rewrite it in that vocabulary
if it does not. Prefer the words someone produces while the rule applies,
including the rationalisation they would be drafting to talk themselves out
of it. See create_rule for the full argument and the measurement behind it.
of it — that rationalisation is often the only text in existence at the
moment the rule should fire:
RETRIEVES: "catching yourself drafting 'this is small enough to not
count' about a rule you have already read"
COLLAPSES: "when the next action would conflict with a standing rule"
See create_rule for the full argument and the measurement behind it.
TO EMPTY A FIELD, NAME IT: clear_fields=["verify_with"]. Passing "" cannot
do it — "" means "leave this alone" here, which is what lets you update
@@ -716,13 +730,15 @@ async def create_preference(
session would actually be producing then: the command it is about to run,
the code it is writing, the thing the operator just asked for.
Show the moment rather than classifying it. Write "the operator pasted a stack
trace and said it is still broken" rather than "during hard debugging";
write "about to `git push` with an earlier CI run still unread" rather
than "when pacing actions". The second spelling of each is a category, and
no session ever produces a category — it produces the command, the error
text, the half-formed ask. A trigger naming the abstraction collapses the
record toward its title and it never arrives.
Show the moment rather than classifying it:
RETRIEVES: "the operator pasted a stack trace and said it is still
broken"
COLLAPSES: "during hard debugging"
The second is a category, and no session ever produces a category — it
produces the command, the error text, the half-formed ask. A trigger
naming the abstraction collapses the record toward its title and it
never arrives. Both of those describe the same preference; only one of
them can be found at the moment it applies.
`arose_from_id` IS REQUIRED for the same kind of reason. A preference is
expected to change as the work teaches it, and a corpus that drifts with
@@ -826,7 +842,11 @@ async def update_preference(
operator's message that should have summoned it. If it describes a
situation rather than quoting the moment, rewrite it in the words they
actually type — and in the words YOU would be producing while about to get
this wrong. update_rule carries the full argument.
this wrong:
RETRIEVES: "the operator said 'clean this up' or 'make it work like',
naming an outcome rather than a change"
COLLAPSES: "when reading any request from the operator"
update_rule carries the full argument.
Empty strings leave fields unchanged; clear_fields empties them by name,
exactly as update_rule does.
+73 -33
View File
@@ -1,4 +1,4 @@
"""Every rule-write surface documents the trigger it can write (#3855).
"""Every rule-write surface SHOWS what a trigger looks like (#3855).
WHY THIS EXISTS
@@ -19,10 +19,45 @@ trigger that already reads fine as English is the one nobody rewrites.
WHAT THIS PINS
One structural property, no wording: a tool that accepts `when_to_apply`
mentions it in its docstring. That is exactly what `update_preference`
failed before #3855 — the parameter existed and the docstring never said so,
which is how six triggers got written against no contract at all.
Two properties, neither of them wording:
1. A tool accepting `when_to_apply` mentions it. Exactly what
`update_preference` failed before #3855 — the parameter existed and
the docstring never said so.
2. It carries a worked CONTRAST: one `RETRIEVES:` example and one
`COLLAPSES:` example, quoted, and different from each other.
WHY A CONVENTION RATHER THAN A PROSE HEURISTIC
The second property guards the softer regression — guidance kept but
abstracted back to "name the moment in session vocabulary", which is advice
about being concrete that is not itself concrete, and is the shape that was
already on file while the corpus filled with categories.
Two attempts to detect that in free prose were written and discarded, and
the reason is worth keeping because it generalises:
- Counting quoted multi-word phrases anywhere in the docstring measured
ambient quotation, not demonstrated triggers. It PASSED the abstracted
version by scoring unrelated prose, and text with an odd number of
quote characters produced matches spanning the gap BETWEEN two
unrelated phrases.
- Scoping that count to a window after each "trigger"/"when_to_apply"
mention then FAILED `create_preference` in its correct state, because
its examples sit further from the first mention than any defensible
window reaches.
Both were proxies trying to infer demonstration from prose. The fix was to
stop inferring: a two-line labelled contrast is unambiguous to parse, free
in its wording, and a better teaching form than the sentences it replaced —
`RETRIEVES:`/`COLLAPSES:` names the mechanism, so the label does work for
the reader instead of only for the test. Where a property cannot be
measured, changing the shape of the thing is cheaper than a cleverer
measurement.
What it still cannot do is judge whether a GOOD example is good. It catches
what actually happened: a write surface shipping with the contrast absent,
half-present, or tidied back into abstract advice.
THE SURFACE LIST IS DERIVED, NEVER HAND-KEPT. It comes from what
`rulebooks.register()` actually hands the server, filtered to signatures
@@ -30,34 +65,9 @@ taking `when_to_apply`, so a rule-write tool added later is in scope on the
day it is added. A hand-kept list has to be remembered by the person least
likely to know it exists — the argument `RANKED_SOURCES` makes in
services/rule_usage.py, and the reason #3855 happened at all.
WHAT THIS DELIBERATELY DOES NOT PIN, AND WHY IT IS NOT HERE
The regression worth catching is the one above's *softer* twin: guidance
kept but abstracted back to "name the moment in session vocabulary", which
is advice about being concrete that is not itself concrete — the shape that
was already on file while the corpus filled with categories.
Two predicates for it were written and both were discarded, because each was
falsified and each failed:
- Counting quoted multi-word phrases anywhere in the docstring measured
ambient quotation, not demonstrated triggers. It passed the broken
version by scoring unrelated prose ("what happens if someone doesn't do
this"), and prose with an odd number of quote characters produced
matches spanning the gap BETWEEN two unrelated phrases.
- Scoping that count to a window after each "trigger"/"when_to_apply"
mention then failed `create_preference` in its CORRECT state, because
its examples sit further from the first mention than any defensible
window reaches.
Rule 167's standard settles it: a check that passes on the broken code reads
as coverage and stops anyone looking again, so no check is the better of the
two. Whether a docstring teaches the shape WELL stays a reading judgment,
and the record of that decision lives here so the next author does not spend
the same two rounds discovering it.
"""
import inspect
import re
import pytest
@@ -65,6 +75,11 @@ from tests.helpers import tool_doc as _doc
_MODULE = "scribe.mcp.tools.rulebooks"
# The worked contrast. Quoted so the example's own punctuation stays inside
# it, labelled so no proximity guess is needed to find it.
_RETRIEVES = re.compile(r'RETRIEVES:\s*"([^"]+)"')
_COLLAPSES = re.compile(r'COLLAPSES:\s*"([^"]+)"')
def _registered_tools() -> list:
"""Every function `rulebooks.register()` actually hands to the server.
@@ -111,6 +126,31 @@ def test_a_trigger_writing_surface_documents_the_field(name):
"docstring. The tool docstring is the agent-facing contract "
"(rule 119), and a trigger written against no contract is the #3855 "
"defect: six preferences named a category instead of a moment, and "
"a category is not a thing any session ever types. Say what the "
"field is for, and show a moment — see create_rule."
"a category is not a thing any session ever types."
)
@pytest.mark.parametrize("name", _trigger_writers())
def test_a_trigger_writing_surface_shows_the_contrast(name):
"""Advice about being concrete has to be concrete itself."""
doc = _doc(_MODULE, name)
retrieves, collapses = _RETRIEVES.findall(doc), _COLLAPSES.findall(doc)
assert retrieves and collapses, (
f"{name} is missing its worked contrast — found "
f"{len(retrieves)} RETRIEVES and {len(collapses)} COLLAPSES example(s), "
"and one of each is required. Telling an author to 'name the moment "
"in session vocabulary' without showing one is the guidance that was "
"already on file while the corpus filled with categories. Add two "
'quoted lines: RETRIEVES: "<a moment a session actually produces — a '
'command, an error, a half-formed ask>" and COLLAPSES: "<the same '
'thing named as a category>".'
)
same = set(retrieves) & set(collapses)
assert not same, (
f"{name} shows the same text as both RETRIEVES and COLLAPSES "
f"({next(iter(same))!r}). The pair teaches by DIFFERING — one moment "
"written the way a session would produce it, and the same moment "
"written as a category. Identical halves demonstrate nothing."
)