feat(plugin): a high-confidence rule is put in front of a command, not beside its result (#4214)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / integration (push) Successful in 59s
CI & Build / Python tests (push) Failing after 1m13s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / integration (push) Successful in 59s
CI & Build / Python tests (push) Failing after 1m13s
CI & Build / Build & push image (push) Skipped
Milestone 419 step 3, the pre-act checkpoint. Every rule surface in this plugin returns `additionalContext`, which Claude Code delivers alongside the tool RESULT — so the rule is read after the call is written and lands as commentary on a decision already made. That is the milestone's central finding, measured over a session with seven misses, three caught by the operator and none by this system. The action arm can now return a `deny` instead. The act does not run, the rule's text can be read before the call exists, and the remedy is one `get_rule` call after which the act may be re-submitted unchanged. Nothing reaches the operator: a deny is a message to the model. "CONSEQUENTIAL" IS DERIVED, NOT ENUMERATED. The obvious implementation lists act kinds — a write to product code, a schema change, a bulk classification, a merge. Every one of those is consequential because THIS operator wrote rules about it, and shipping that list is this instance's corpus hard-coded into the product (rule 115). So the corpus decides: an act is consequential when the install's own rules speak to it above the checkpoint bar. A fresh install with no rules never stops anything. FOUR CONDITIONS, EACH PREVENTING A DIFFERENT WRONG. Above the bar; a rule and never a preference (which claims no such force); the band's top hit only (the ranker's confidence claim attaches to its first element); and only a rule the session has NOT opened — `held` is observable from the get_rule PostToolUse hook (#4100), not self-report. WHY "NOT OPENED" RATHER THAN "NO OUTCOME RECORDED". An outcome can be satisfied with one cheap call asserting compliance without producing any, and a checkpoint dismissible that way manufactures exactly the compliance data step 2 was built to measure. Reading a rule cannot be faked in that direction: after `get_rule` the statement is in context, which is the whole of what was wanted. THE BAR IS MEASURED. `retrieval_telemetry(days=30)`: write_path_rule p90 0.7628 max 0.8817; pre_tool_rule p90 0.7373 max 0.8293. 0.80 is above p90 on both and below max on both, so it selects from the top decile of an already selective arm and is still reachable. It ships as a setting with a Settings card, because a cosine distance in one model's geometry over one corpus cannot transfer. TWO GUARDS ON THE WORST CASE: at most one hold per rule and five per session, so a mis-set floor degrades to a noisy session rather than one that cannot proceed. The ledger lives in the swept directory and is named `.ids`, so the existing compaction-clear guards cover it. WRITES ARE NOT HELD, AND THAT IS THE OPERATOR'S DECISION RATHER THAN MINE. `scribe_prior_art.sh` carries a tested property that it never returns a permissionDecision — a recall aid may not stand in the way of a write. Three of the milestone's seven misses were file edits and none are reachable from the command side, so there is a live argument for extending this; that argument is exactly why the boundary is now asserted by a test rather than left to memory. The write-path arm computes and returns the same block so the decision can be revisited with evidence; the hook ignores it, and a change of mind is a hook edit rather than a feature. Verified by lifting `checkpoint_for` and `_rule_band` out of source with `ast` and exercising the shipped functions over 17 populations, by running the ledger and deny envelope in bash (10 cases, including that a refused hold is not written and that a garbled rule id fails closed), and by scripts/check_plugin.py — which caught the unminted plugin version, 0300 -> 0426. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -87,6 +87,7 @@ const kbWritePathEnabled = ref(true);
|
||||
// unrelated code through (#2223). Shares top-k, not the threshold.
|
||||
const kbWritePathThreshold = ref("0.68");
|
||||
const kbRuleHintThreshold = ref("0.72");
|
||||
const kbCheckpointThreshold = ref("0.80");
|
||||
// The two ACT arms no longer share a bar (#3853). A write-path query is a code
|
||||
// payload; a pre-tool query is a shell command, often under a dozen words —
|
||||
// less text, less signal, lower scores for the same relevance. Measured at one
|
||||
@@ -275,6 +276,10 @@ async function saveKbInject() {
|
||||
// Bash call, so a fallback of 0 would put a rule in front of every command.
|
||||
const trT = Math.min(1, Math.max(0, Number(kbToolRuleThreshold.value) || 0.68));
|
||||
const prT = Math.min(1, Math.max(0, Number(kbPromptRuleThreshold.value) || 0.72));
|
||||
// The checkpoint bar, and the `|| default` guard matters most here of all:
|
||||
// this is the only number that can STOP a call, so a fallback of 0 would
|
||||
// hold the first command of every session behind whatever ranked first.
|
||||
const cpT = Math.min(1, Math.max(0, Number(kbCheckpointThreshold.value) || 0.8));
|
||||
const rpT = Math.min(1, Math.max(0, Number(kbReportPrefThreshold.value) || 0.72));
|
||||
// The budgets, clamped the way the server clamps them: a whole number in
|
||||
// [1, 10]. Never 0 — an arm turned off is turned off by its switch, and a
|
||||
@@ -300,6 +305,7 @@ async function saveKbInject() {
|
||||
kbPlanMatchThreshold.value = String(planT);
|
||||
kbWritePathThreshold.value = String(wpT);
|
||||
kbRuleHintThreshold.value = String(rhT);
|
||||
kbCheckpointThreshold.value = String(cpT);
|
||||
kbToolRuleThreshold.value = String(trT);
|
||||
kbPromptRuleThreshold.value = String(prT);
|
||||
kbReportPrefThreshold.value = String(rpT);
|
||||
@@ -319,6 +325,12 @@ async function saveKbInject() {
|
||||
// in services/plugin_context.py for why rules cannot share the
|
||||
// code threshold any more than code could share the prose one.
|
||||
kb_rulehint_threshold: String(rhT),
|
||||
// The one bar that stops a call rather than annotating it. Its own key,
|
||||
// never derived from the two rule bars above: it answers a different
|
||||
// question of the same scores — not "is this worth showing" but "is the
|
||||
// corpus confident enough to be read first" — and a number that moves
|
||||
// when another moves is a number nobody can reason about.
|
||||
kb_checkpoint_threshold: String(cpT),
|
||||
// A FOURTH and FIFTH bar, and they are separate keys on purpose: the
|
||||
// whole finding of #3853 is that one number cannot serve arms whose
|
||||
// queries are different shapes. Moving one must not move the others.
|
||||
@@ -790,6 +802,9 @@ onMounted(async () => {
|
||||
kbInjectTopK.value = allSettings.kb_autoinject_top_k;
|
||||
}
|
||||
kbWritePathEnabled.value = allSettings.kb_writepath_enabled !== "false";
|
||||
if (allSettings.kb_checkpoint_threshold !== undefined) {
|
||||
kbCheckpointThreshold.value = allSettings.kb_checkpoint_threshold;
|
||||
}
|
||||
if (allSettings.kb_rulehint_threshold !== undefined) {
|
||||
kbRuleHintThreshold.value = allSettings.kb_rulehint_threshold;
|
||||
}
|
||||
@@ -1723,6 +1738,32 @@ async function deleteUser(userId: number) {
|
||||
/>
|
||||
<p class="field-hint">How many standing rules one edit may be shown (1–10).</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-checkpoint-threshold">Hold-before-acting threshold (0–1)</label>
|
||||
<input
|
||||
id="kb-checkpoint-threshold"
|
||||
v-model="kbCheckpointThreshold"
|
||||
type="number"
|
||||
min="0"
|
||||
max="1"
|
||||
step="0.01"
|
||||
class="fs-input input"
|
||||
style="max-width: 8rem"
|
||||
/>
|
||||
<p class="field-hint">
|
||||
Every hint above arrives <em>beside</em> the result of the action it
|
||||
was about — useful to read afterwards, too late to change what the
|
||||
action was. Above this bar a standing rule is instead put
|
||||
<em>in front</em> of the command: Claude is asked to read the rule
|
||||
first, and then runs the command anyway if the rule does not apply.
|
||||
Set well above the thresholds above, because it costs a round trip
|
||||
rather than a line. Only standing rules can hold a command, never
|
||||
preferences; only a rule this session has not already read; and
|
||||
never more than once per rule or five times per session, so a bar
|
||||
set too low is a chatty session rather than a stuck one. Commands
|
||||
only — file edits are never held.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-toolrule-threshold">Command confidence threshold (0–1)</label>
|
||||
<input
|
||||
|
||||
Reference in New Issue
Block a user