feat(plugin): a recognized retrieval miss has a route, and the record comes first (#4133)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m34s
CI & Build / Build & push image (push) Successful in 43s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m34s
CI & Build / Build & push image (push) Successful in 43s
The tuning loop shipped in steps 4 and 6 and logged zero events in its lifetime. `tune_retrieval`, `retrieval_telemetry` and `retrieval_surfaces` appeared on no instruction surface at all — not the skills, not the hooks, not the MCP instructions — so the decision that "the model should be the thing handling it 9 times out of 10" could not begin to happen. What was missing was not an auditor but a route. `using-scribe` now carries it, ordered: read the refused records, fix the trigger, and only then consider the dial. The order is the content. A rule's `when_to_apply` IS the text its score is computed against, so a miss is evidence about that text first; rewording one trigger changes one rule's reach, while moving a floor changes what every record on the surface does and cannot tell a badly-worded trigger from a genuinely distant one. Measured, and the reason the order is asserted rather than suggested: rule 1 scored 0.6515 and ranked 5th for the moment it governed, behind three rules that restrained the same act. Every percentile said "lower the floor"; at 0.60 the arm delivered those three restraints and still not rule 1. Rewriting the trigger to lead with the symptom put it 1st at 0.7130. Also: the create path gets a precondition. A new record is itself a retrieval-affecting act, so before writing one, what_might_apply asks what already covers that moment — fifty candidates and no bar, because a bar is what lets the existing record hide. `_INSTRUCTIONS` gets one index line, not the route: 1,986 of 2,000 characters, since Claude Code cuts the rest mid-word (#2562). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -318,6 +318,19 @@ with no trigger is not a quiet rule, it is an unreachable one. Write the moment
|
||||
in the words a session actually produces — the command, the error, the
|
||||
half-formed ask — not the category it belongs to.
|
||||
|
||||
**Before writing one, ask what already covers that moment.**
|
||||
`what_might_apply("the moment you are about to write a record for")` — fifty
|
||||
candidates and no bar, so an existing record cannot hide under a threshold the
|
||||
way it can from `search`. When something already covers the moment, the reach
|
||||
is that record: improve its trigger or its statement rather than standing a
|
||||
second one beside it. Two records describing the same moment compete in one
|
||||
ranked list against one budget, and the slot they take from each other is the
|
||||
third candidate that would have said something different. Two records may
|
||||
legitimately share a moment and say *different* things — a rule for what must
|
||||
happen, a preference for how to report it. What this catches is the same thing
|
||||
said twice at two strengths, which is worse than either alone: a session that
|
||||
retrieves the softer copy has been told that binding guidance is optional.
|
||||
|
||||
**First ask whether it's a rule at all.** A rule is prose you have to remember
|
||||
and apply; Scribe's other entities are structure a tool can resolve and check.
|
||||
Visual standards belong in a **design system**, not a rulebook — a token can be
|
||||
@@ -326,6 +339,49 @@ and none of that survives being written as a rule. A repeatable procedure is a
|
||||
**process**; reusable code is a **snippet**. Reach for a rule when the thing
|
||||
really is a standing instruction about how to work.
|
||||
|
||||
## When a record doesn't reach the moment it should
|
||||
|
||||
Retrieval misjudging is ordinary, and it is fixable — but only by whoever
|
||||
notices. **Either direction counts:** a rule that should have governed a moment
|
||||
and never arrived, and a rule that arrives on every turn and never applies. So
|
||||
does **either noticer**: the operator saying *"that should have fired"*, and you
|
||||
noticing it yourself — you reached for a rule nobody offered you, or you were
|
||||
handed the same rule five times and set it aside five times.
|
||||
|
||||
**Take it to the record first and the dial second.** A rule's `when_to_apply`
|
||||
IS the text its similarity score is computed against, so when a rule misses a
|
||||
moment it governs, the overwhelmingly likely cause is that its trigger does not
|
||||
describe that moment in the words a session actually produces. Rewording one
|
||||
trigger changes one rule's reach. Moving a floor changes what every record on
|
||||
that surface does, and a floor cannot tell a badly-worded trigger from a
|
||||
genuinely distant record — so one lowered to rescue a single rule admits
|
||||
everything else that was sitting in the same band.
|
||||
|
||||
1. **Read the refused records.** `retrieval_telemetry(days=N,
|
||||
near_miss_samples=5)` names by id what each surface refused and by how much.
|
||||
Open them with `get_rule` / `get_note`. This is the step that carries the
|
||||
answer: the statistic says a record was close, and only the record says
|
||||
whether it was *right*.
|
||||
2. **Fix the trigger.** `update_rule(when_to_apply=...)`, written as the
|
||||
symptom — what the session was doing or saying at the moment it needed this
|
||||
rule — not the situation the rule belongs to. Then check that it worked:
|
||||
`what_might_apply("the moment, in the operator's own words")` and read where
|
||||
the rule now ranks. The change is measurable, so measure it, and say the
|
||||
before and after when you report it.
|
||||
3. **Then consider the dial.** `retrieval_surfaces` shows what is in force per
|
||||
arm and whether the number is still calibrated; `tune_retrieval` moves it.
|
||||
`reason` is required and has to say what you read, because it is what lets
|
||||
the operator disagree with a number they did not choose.
|
||||
|
||||
Reaching for `tune_retrieval` before opening a single record is the wrong move,
|
||||
and it is the one that feels efficient. Worked example, measured on this
|
||||
install: a rule granting a routine push scored 0.6515 and ranked 5th for the
|
||||
moment it governed, behind three rules that *restrained* the same act. Every
|
||||
percentile said "lower the floor" — and lowering it would have delivered those
|
||||
three restraints and still not the rule. Rewriting the trigger to lead with the
|
||||
symptom moved the same rule to 1st at 0.7130, ahead of all three. Only then was
|
||||
the floor worth touching.
|
||||
|
||||
## Building UI: the project's design system binds
|
||||
|
||||
`enter_project` names the project's `design_system` when it has one. Before
|
||||
|
||||
Reference in New Issue
Block a user