feat(plugin): a high-confidence rule is put in front of a command, not beside its result (#4214)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / integration (push) Successful in 59s
CI & Build / Python tests (push) Failing after 1m13s
CI & Build / Build & push image (push) Skipped

Milestone 419 step 3, the pre-act checkpoint. Every rule surface in this
plugin returns `additionalContext`, which Claude Code delivers alongside the
tool RESULT — so the rule is read after the call is written and lands as
commentary on a decision already made. That is the milestone's central
finding, measured over a session with seven misses, three caught by the
operator and none by this system.

The action arm can now return a `deny` instead. The act does not run, the
rule's text can be read before the call exists, and the remedy is one
`get_rule` call after which the act may be re-submitted unchanged. Nothing
reaches the operator: a deny is a message to the model.

"CONSEQUENTIAL" IS DERIVED, NOT ENUMERATED. The obvious implementation lists
act kinds — a write to product code, a schema change, a bulk classification, a
merge. Every one of those is consequential because THIS operator wrote rules
about it, and shipping that list is this instance's corpus hard-coded into the
product (rule 115). So the corpus decides: an act is consequential when the
install's own rules speak to it above the checkpoint bar. A fresh install with
no rules never stops anything.

FOUR CONDITIONS, EACH PREVENTING A DIFFERENT WRONG. Above the bar; a rule and
never a preference (which claims no such force); the band's top hit only (the
ranker's confidence claim attaches to its first element); and only a rule the
session has NOT opened — `held` is observable from the get_rule PostToolUse
hook (#4100), not self-report.

WHY "NOT OPENED" RATHER THAN "NO OUTCOME RECORDED". An outcome can be
satisfied with one cheap call asserting compliance without producing any, and
a checkpoint dismissible that way manufactures exactly the compliance data
step 2 was built to measure. Reading a rule cannot be faked in that direction:
after `get_rule` the statement is in context, which is the whole of what was
wanted.

THE BAR IS MEASURED. `retrieval_telemetry(days=30)`: write_path_rule p90
0.7628 max 0.8817; pre_tool_rule p90 0.7373 max 0.8293. 0.80 is above p90 on
both and below max on both, so it selects from the top decile of an already
selective arm and is still reachable. It ships as a setting with a Settings
card, because a cosine distance in one model's geometry over one corpus cannot
transfer.

TWO GUARDS ON THE WORST CASE: at most one hold per rule and five per session,
so a mis-set floor degrades to a noisy session rather than one that cannot
proceed. The ledger lives in the swept directory and is named `.ids`, so the
existing compaction-clear guards cover it.

WRITES ARE NOT HELD, AND THAT IS THE OPERATOR'S DECISION RATHER THAN MINE.
`scribe_prior_art.sh` carries a tested property that it never returns a
permissionDecision — a recall aid may not stand in the way of a write. Three
of the milestone's seven misses were file edits and none are reachable from
the command side, so there is a live argument for extending this; that
argument is exactly why the boundary is now asserted by a test rather than
left to memory. The write-path arm computes and returns the same block so the
decision can be revisited with evidence; the hook ignores it, and a change of
mind is a hook edit rather than a feature.

Verified by lifting `checkpoint_for` and `_rule_band` out of source with `ast`
and exercising the shipped functions over 17 populations, by running the
ledger and deny envelope in bash (10 cases, including that a refused hold is
not written and that a garbled rule id fails closed), and by
scripts/check_plugin.py — which caught the unminted plugin version, 0300 ->
0426.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-21 00:26:54 -04:00
co-authored by Claude Opus 5
parent bb8013928f
commit 91bc0fb01e
7 changed files with 685 additions and 4 deletions
+41
View File
@@ -87,6 +87,7 @@ const kbWritePathEnabled = ref(true);
// unrelated code through (#2223). Shares top-k, not the threshold.
const kbWritePathThreshold = ref("0.68");
const kbRuleHintThreshold = ref("0.72");
const kbCheckpointThreshold = ref("0.80");
// The two ACT arms no longer share a bar (#3853). A write-path query is a code
// payload; a pre-tool query is a shell command, often under a dozen words —
// less text, less signal, lower scores for the same relevance. Measured at one
@@ -275,6 +276,10 @@ async function saveKbInject() {
// Bash call, so a fallback of 0 would put a rule in front of every command.
const trT = Math.min(1, Math.max(0, Number(kbToolRuleThreshold.value) || 0.68));
const prT = Math.min(1, Math.max(0, Number(kbPromptRuleThreshold.value) || 0.72));
// The checkpoint bar, and the `|| default` guard matters most here of all:
// this is the only number that can STOP a call, so a fallback of 0 would
// hold the first command of every session behind whatever ranked first.
const cpT = Math.min(1, Math.max(0, Number(kbCheckpointThreshold.value) || 0.8));
const rpT = Math.min(1, Math.max(0, Number(kbReportPrefThreshold.value) || 0.72));
// The budgets, clamped the way the server clamps them: a whole number in
// [1, 10]. Never 0 — an arm turned off is turned off by its switch, and a
@@ -300,6 +305,7 @@ async function saveKbInject() {
kbPlanMatchThreshold.value = String(planT);
kbWritePathThreshold.value = String(wpT);
kbRuleHintThreshold.value = String(rhT);
kbCheckpointThreshold.value = String(cpT);
kbToolRuleThreshold.value = String(trT);
kbPromptRuleThreshold.value = String(prT);
kbReportPrefThreshold.value = String(rpT);
@@ -319,6 +325,12 @@ async function saveKbInject() {
// in services/plugin_context.py for why rules cannot share the
// code threshold any more than code could share the prose one.
kb_rulehint_threshold: String(rhT),
// The one bar that stops a call rather than annotating it. Its own key,
// never derived from the two rule bars above: it answers a different
// question of the same scores — not "is this worth showing" but "is the
// corpus confident enough to be read first" — and a number that moves
// when another moves is a number nobody can reason about.
kb_checkpoint_threshold: String(cpT),
// A FOURTH and FIFTH bar, and they are separate keys on purpose: the
// whole finding of #3853 is that one number cannot serve arms whose
// queries are different shapes. Moving one must not move the others.
@@ -790,6 +802,9 @@ onMounted(async () => {
kbInjectTopK.value = allSettings.kb_autoinject_top_k;
}
kbWritePathEnabled.value = allSettings.kb_writepath_enabled !== "false";
if (allSettings.kb_checkpoint_threshold !== undefined) {
kbCheckpointThreshold.value = allSettings.kb_checkpoint_threshold;
}
if (allSettings.kb_rulehint_threshold !== undefined) {
kbRuleHintThreshold.value = allSettings.kb_rulehint_threshold;
}
@@ -1723,6 +1738,32 @@ async function deleteUser(userId: number) {
/>
<p class="field-hint">How many standing rules one edit may be shown (110).</p>
</div>
<div class="field">
<label for="kb-checkpoint-threshold">Hold-before-acting threshold (01)</label>
<input
id="kb-checkpoint-threshold"
v-model="kbCheckpointThreshold"
type="number"
min="0"
max="1"
step="0.01"
class="fs-input input"
style="max-width: 8rem"
/>
<p class="field-hint">
Every hint above arrives <em>beside</em> the result of the action it
was about — useful to read afterwards, too late to change what the
action was. Above this bar a standing rule is instead put
<em>in front</em> of the command: Claude is asked to read the rule
first, and then runs the command anyway if the rule does not apply.
Set well above the thresholds above, because it costs a round trip
rather than a line. Only standing rules can hold a command, never
preferences; only a rule this session has not already read; and
never more than once per rule or five times per session, so a bar
set too low is a chatty session rather than a stuck one. Commands
only — file edits are never held.
</p>
</div>
<div class="field">
<label for="kb-toolrule-threshold">Command confidence threshold (01)</label>
<input