diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index a266b4f..35f110d 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "scribe", "description": "Scribe for Claude Code: connects the scribe MCP server, adds the hooks that deliver live project state and relevant records at the right moment, ships the shared client-neutral Scribe skills (using-scribe, writing-plans, reporting-back, systematic-debugging, verification, brainstorming, reusing-code, shape-accounting), and syncs your saved Scribe Processes as skills (/scribe:sync).", - "version": "2026.09.17.1618", + "version": "2026.09.18.0436", "author": { "name": "Bryan Van Deusen" }, diff --git a/plugin/skills/using-scribe/SKILL.md b/plugin/skills/using-scribe/SKILL.md index ea83b4b..526326d 100644 --- a/plugin/skills/using-scribe/SKILL.md +++ b/plugin/skills/using-scribe/SKILL.md @@ -318,6 +318,19 @@ with no trigger is not a quiet rule, it is an unreachable one. Write the moment in the words a session actually produces — the command, the error, the half-formed ask — not the category it belongs to. +**Before writing one, ask what already covers that moment.** +`what_might_apply("the moment you are about to write a record for")` — fifty +candidates and no bar, so an existing record cannot hide under a threshold the +way it can from `search`. When something already covers the moment, the reach +is that record: improve its trigger or its statement rather than standing a +second one beside it. Two records describing the same moment compete in one +ranked list against one budget, and the slot they take from each other is the +third candidate that would have said something different. Two records may +legitimately share a moment and say *different* things — a rule for what must +happen, a preference for how to report it. What this catches is the same thing +said twice at two strengths, which is worse than either alone: a session that +retrieves the softer copy has been told that binding guidance is optional. + **First ask whether it's a rule at all.** A rule is prose you have to remember and apply; Scribe's other entities are structure a tool can resolve and check. Visual standards belong in a **design system**, not a rulebook — a token can be @@ -326,6 +339,49 @@ and none of that survives being written as a rule. A repeatable procedure is a **process**; reusable code is a **snippet**. Reach for a rule when the thing really is a standing instruction about how to work. +## When a record doesn't reach the moment it should + +Retrieval misjudging is ordinary, and it is fixable — but only by whoever +notices. **Either direction counts:** a rule that should have governed a moment +and never arrived, and a rule that arrives on every turn and never applies. So +does **either noticer**: the operator saying *"that should have fired"*, and you +noticing it yourself — you reached for a rule nobody offered you, or you were +handed the same rule five times and set it aside five times. + +**Take it to the record first and the dial second.** A rule's `when_to_apply` +IS the text its similarity score is computed against, so when a rule misses a +moment it governs, the overwhelmingly likely cause is that its trigger does not +describe that moment in the words a session actually produces. Rewording one +trigger changes one rule's reach. Moving a floor changes what every record on +that surface does, and a floor cannot tell a badly-worded trigger from a +genuinely distant record — so one lowered to rescue a single rule admits +everything else that was sitting in the same band. + +1. **Read the refused records.** `retrieval_telemetry(days=N, + near_miss_samples=5)` names by id what each surface refused and by how much. + Open them with `get_rule` / `get_note`. This is the step that carries the + answer: the statistic says a record was close, and only the record says + whether it was *right*. +2. **Fix the trigger.** `update_rule(when_to_apply=...)`, written as the + symptom — what the session was doing or saying at the moment it needed this + rule — not the situation the rule belongs to. Then check that it worked: + `what_might_apply("the moment, in the operator's own words")` and read where + the rule now ranks. The change is measurable, so measure it, and say the + before and after when you report it. +3. **Then consider the dial.** `retrieval_surfaces` shows what is in force per + arm and whether the number is still calibrated; `tune_retrieval` moves it. + `reason` is required and has to say what you read, because it is what lets + the operator disagree with a number they did not choose. + +Reaching for `tune_retrieval` before opening a single record is the wrong move, +and it is the one that feels efficient. Worked example, measured on this +install: a rule granting a routine push scored 0.6515 and ranked 5th for the +moment it governed, behind three rules that *restrained* the same act. Every +percentile said "lower the floor" — and lowering it would have delivered those +three restraints and still not the rule. Rewriting the trigger to lead with the +symptom moved the same rule to 1st at 0.7130, ahead of all three. Only then was +the floor worth touching. + ## Building UI: the project's design system binds `enter_project` names the project's `design_system` when it has one. Before diff --git a/src/scribe/mcp/server.py b/src/scribe/mcp/server.py index 7462c89..9e876a1 100644 --- a/src/scribe/mcp/server.py +++ b/src/scribe/mcp/server.py @@ -49,6 +49,8 @@ client reads Agent Skills) and in each tool's description. The index: candidates, no bar. search(content_type="rule") reads one you already suspect. Silence means nothing matched, not none. Rules bind; preferences guide. +- MISSED: a rule that should have reached you is a trigger to fix, not a + floor to move (retrieval_telemetry). - RECALL: search before acting, scoped with the active project_id. - RECORD: create_task; a fix is kind="issue". add_task_log as you go; status in_progress on start, done on finish. Tag system_ids as you write. diff --git a/tests/test_guidance_ownership.py b/tests/test_guidance_ownership.py index 43140cc..656a6c3 100644 --- a/tests/test_guidance_ownership.py +++ b/tests/test_guidance_ownership.py @@ -142,6 +142,15 @@ TOPICS: tuple[Topic, ...] = ( Topic("where a new rule goes, and its trigger", U, ("create_project_rule", "when_to_apply"), "whichever home it gets"), Topic("a rule vs the other entities", U, ("standing instruction",), "first ask whether it's a rule at all"), + # Milestone 416 step 9: the tuning tools shipped in #4102/#4104 and were + # named on NO instruction surface — measured, `retrieval_tuning_history` + # returned zero events. Machinery with no route to it. + Topic("a missed rule is a trigger to fix before a floor to move", U, + ("retrieval_telemetry", "tune_retrieval", "retrieval_surfaces"), + "take it to the record first and the dial second", + index=("retrieval_telemetry",)), + Topic("ask what already covers a moment before writing a record", U, + ("what_might_apply",), "ask what already covers that moment"), Topic("reference notes update in place; dev-logs don't", U, ("reference note",), "state updates in place; chronicles don't"), # ── process arcs — owned by their skills ── diff --git a/tests/test_retrieval_miss_route.py b/tests/test_retrieval_miss_route.py new file mode 100644 index 0000000..48e8fcf --- /dev/null +++ b/tests/test_retrieval_miss_route.py @@ -0,0 +1,158 @@ +"""A recognized retrieval miss has a route, and the route is ordered (#4133). + +WHY THIS EXISTS + +Milestone 416 built the tuning loop across steps 4 and 6 — `retrieval_surfaces` +to read a dial, `tune_retrieval` to move one with its argument attached, +`migrate_retrieval_floor` to carry it across a model change — and then measured, +on 2026-09-17, that `retrieval_tuning_history` returned **zero events**. Not one +call, ever. + +The cause was not a tool being wrong. The three tools appeared on NO instruction +surface: not the skills, not the hooks, not the MCP `_INSTRUCTIONS`. They were +classified in `server.py`'s read/write lists and named nowhere a session would +read them. The operator's decision for #4102 — *"the model should be the thing +handling it 9 times out of 10"* — could not begin to happen. + +So this file guards a ROUTE, not a behaviour: that a session which notices a +rule missed the moment it governed has somewhere written down to take it. + +WHY THE ORDER IS THE LOAD-BEARING PART + +`test_the_record_comes_before_the_dial` is the test that matters here, and it is +positional on purpose. Both halves of the route are individually reasonable — +read the telemetry, move the floor — and stated in either order they read as a +menu. They are not a menu. A rule's `when_to_apply` IS the text its score is +computed against, so a miss is evidence about that text first; a floor is one +number governing every record on the surface and cannot tell a badly-worded +trigger from a genuinely distant one. + +Measured on this install, which is why the order is asserted rather than +suggested: rule 1 "`dev` is home" scored 0.6515 and ranked 5th for the moment it +governed, behind three rules that RESTRAINED the same act. Every percentile said +"lower the floor" — and at 0.60 the arm delivered those three restraints and +still not rule 1. Rewriting the trigger to lead with the symptom put the same +rule 1st at 0.7130. The dial-first path was available, statistically supported, +and wrong. +""" +from __future__ import annotations + +import pathlib +import re + +ROOT = pathlib.Path(__file__).resolve().parents[1] +SKILL = ROOT / "plugin/skills/using-scribe/SKILL.md" + +# The three tools that shipped with no route to them. Named together because +# the gap was all three at once, and a partial fix would leave the loop broken +# at whichever step lost its pointer. +TUNING_TOOLS = ("retrieval_telemetry", "update_rule", "retrieval_surfaces", + "tune_retrieval") + + +def _skill() -> str: + return SKILL.read_text() + + +def _instructions() -> str: + src = (ROOT / "src/scribe/mcp/server.py").read_text() + match = re.search(r'_INSTRUCTIONS = """(.*?)"""', src, re.S) + assert match, "server.py no longer defines _INSTRUCTIONS as a literal" + return match.group(1) + + +def test_the_route_names_every_tool_it_depends_on(): + """The measured gap: machinery with no instruction surface pointing at it.""" + text = _skill() + missing = [t for t in TUNING_TOOLS if t not in text] + assert not missing, ( + f"using-scribe no longer names {missing}. These tools reach a session " + f"through this skill and nowhere else — when they were named on no " + f"surface at all, the tuning loop logged zero events in its lifetime." + ) + + +def test_the_record_comes_before_the_dial(): + """THE GUARD. Stated in the wrong order, the route becomes a menu. + + Positional rather than a presence check, because presence is exactly what + a reordering preserves. Asserted on first occurrence: what a reader meets + first is what they act on. + """ + text = _skill() + read_records = text.index("retrieval_telemetry") + fix_trigger = text.index("update_rule(when_to_apply") + move_dial = text.index("tune_retrieval") + + assert read_records < move_dial, ( + "the route reaches tune_retrieval before retrieval_telemetry. Reading " + "the refused records is the step that separates a real miss from a bar " + "doing its job — a percentile cannot, and has been measured pointing " + "the wrong way." + ) + assert fix_trigger < move_dial, ( + "the route reaches tune_retrieval before update_rule. A miss is " + "evidence about the trigger first: rewording one trigger changes one " + "rule's reach, while moving a floor changes every record on the surface." + ) + + +def test_the_route_says_what_a_floor_costs_that_a_trigger_does_not(): + """The REASON for the order, not just the order. + + Without it the ordering is arbitrary and the first session under deadline + will invert it. The asymmetry is the whole argument: one trigger against + every record on the surface. + """ + text = " ".join(_skill().split()).lower() + assert "take it to the record first and the dial second" in text + assert "changes one rule's reach" in text + assert "changes what every record on" in text + + +def test_a_session_can_notice_its_own_miss(): + """Both noticers, or the route only ever fires when the operator complains. + + The self-noticed case is the one that scales — it does not need somebody + watching — and it is the one a surface written only around operator + feedback silently omits. + """ + text = " ".join(_skill().split()).lower() + assert "either noticer" in text + assert "reached for a rule nobody offered you" in text + # Both directions too: a rule that never arrives and one that always does + # are the same misjudgment, and they point at different dials. + assert "either direction counts" in text + + +def test_writing_a_new_record_asks_what_already_covers_the_moment(): + """The create path is retrieval-affecting, and nothing said so. + + A second record on one moment splits a single budget between two claims, + and when the copies differ in force the softer one teaches a session that + binding guidance is optional. `what_might_apply` rather than `search` + because a bar is exactly what lets the existing record hide. + """ + text = " ".join(_skill().split()).lower() + assert "ask what already covers that moment" in text + assert "what_might_apply" in text + assert "no bar" in text + + +def test_the_index_points_at_the_route_without_restating_it(): + """`_INSTRUCTIONS` is the only surface every MCP client gets (decision #4027). + + It indexes; using-scribe states. A client with no Agent Skills support still + learns the ordering exists and which tool opens it. + + Whitespace-flattened before matching. The block is hard-wrapped to fit a + 2,000-character budget, so a phrase straddles a line break the moment + anything before it changes length — #4103 shipped a guard that broke + exactly that way, on text nobody had touched. + """ + text = " ".join(_instructions().split()) + assert "retrieval_telemetry" in text + assert "not a floor to move" in text + # The route itself must NOT be here — 2,000 characters is the whole budget + # and Claude Code cuts the rest mid-word (#2562). + assert "take it to the record first" not in text.lower()