feat(rules): the command arm gets its own bar, measured (#3853)
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / Python lint (push) Successful in 4s
CI & Build / TypeScript typecheck (push) Successful in 1m2s
CI & Build / Python tests (push) Successful in 1m34s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Build & push image (push) Successful in 35s

One threshold served both act arms. The telemetry says they are not the
same problem:

  write_path_rule   2,325 calls, speaks on 37%, near-miss p50 0.6989
  pre_tool_rule    11,768 calls, speaks on  2%, near-miss p50 0.6794

The second is not quiet, it is mute — 11,530 of 11,768 calls said nothing,
with near-miss p90 at 0.7097 against a 0.72 bar. Refused mass piled one
hundredth under the line is what a bar set too high leaves behind, and the
note arms are the control: auto_inject refuses at p90 0.5463, write_path at
0.6738, both far below theirs.

The cause is query shape, not corpus. A write-path query is a code payload,
long and rich — the case 0.72 was calibrated on. A pre-tool query is a shell
command, often under a dozen words: less text, less signal, lower scores for
the same relevance.

MEASURED. Eight replayed queries against the post-#3855 corpus, consequential
acts against innocuous ones:

  0.7571  git push origin dev              consequential
  0.7245  cd ...; git fetch; git add -A    consequential
  0.7193  git pull --rebase origin dev     consequential
  0.6850  docker compose up -d             consequential
  ------------------------------------- 0.68
  0.6735  wc -l src/*.py && date           innocuous
  0.6544  grep -rn useState src/           innocuous
  0.6099  sed -n '120,160p' package.json   innocuous
  0.6056  ls -la && cat README.md          innocuous

At 0.72 three of four consequential acts retrieved nothing, including
`git pull --rebase origin dev`, where rules 153, 1 and 2 all ranked correctly
between 0.7126 and 0.7193 and were all refused.

The separation is 0.0115 wide. That is a direction, not a settled number, and
the comment says so — near_miss_samples on a few days of post-#3855 traffic
is what settles it.

This also corrects an assumption the old comment stated: it argued 0.68 sat
"below where this corpus's noise sits", inferring a higher floor from the
corpus being homogeneous. Measured, the command arm's noise ceiling is 0.6735,
so 0.68 clears it barely rather than sitting under it.

Lowering is safer now than it would have been. Until #3851 this arm had one
slot, so the bar was the only noise control; the band now filters downstream,
so the bar's job shrank and the bar can.

write_path_rule is unchanged — healthy at 0.72 on its own evidence.

Guards: the two bars parse independently, garbage falls back to its OWN
default rather than to the sibling's (which would silently re-merge them),
the command default stays below the write-path default as a direction check,
and each arm both SEARCHES and REPORTS at its own bar. That last one is a
failure the single-bar code could not have had: retrieval_logs.threshold is
what near-miss analysis is read against, so an arm searching at one number
and logging another misreports the refusal and invites moving the bar that
was already right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-11 14:36:18 -04:00
co-authored by Claude Opus 5
parent 40189147d2
commit 690ca0306e
3 changed files with 216 additions and 2 deletions
+57
View File
@@ -481,6 +481,63 @@ async def test_the_two_write_path_bars_are_independent():
assert cfg["rule_threshold"] == 0.61
@pytest.mark.asyncio
async def test_the_two_act_arms_read_independent_rule_bars():
"""#3853's split: the command arm's bar moves without the write path's.
The two act arms shared one key until the telemetry showed them behaving
like different subsystems at the same number — the write-path arm speaking
on 37% of calls against the command arm's 2%, because a code payload is
long and rich where a shell command is short. A config assembler that
reads one key into both fields would silently undo that, and the symptom
would be invisible: both arms would simply agree again.
"""
from scribe.services import plugin_context as pc
stored = {pc.RULEHINT_THRESHOLD_KEY: "0.75", pc.TOOLRULE_THRESHOLD_KEY: "0.61"}
with patch.object(pc, "get_setting",
AsyncMock(side_effect=lambda uid, k, d: stored.get(k, d))):
cfg = await pc.get_writepath_config(1)
assert cfg["rule_threshold"] == 0.75
assert cfg["tool_rule_threshold"] == 0.61
@pytest.mark.asyncio
async def test_a_garbage_command_bar_falls_back_to_its_own_default():
"""Not to 0.0, and not to the write path's default.
Falling back to 0.0 would attach a rule to every Bash call in the session;
falling back to the sibling's default would quietly re-merge the two bars
that #3853 separated, which is the harder failure to see because the arm
keeps working.
"""
from scribe.services import plugin_context as pc
stored = {pc.TOOLRULE_THRESHOLD_KEY: "banana"}
with patch.object(pc, "get_setting",
AsyncMock(side_effect=lambda uid, k, d: stored.get(k, d))):
cfg = await pc.get_writepath_config(1)
assert cfg["tool_rule_threshold"] == pc.TOOLRULE_DEFAULT_THRESHOLD
def test_the_command_bar_defaults_below_the_write_path_bar():
"""A DIRECTION check, like its sibling above, and for the same rule-115
reason: the value is measured against one corpus, the relationship is not.
A shell command carries less text than a code payload and therefore scores
lower for the same relevance — measured at #3853, where three of four
consequential commands retrieved nothing at the shared bar while the
write-path arm was healthy at it. Tuning either value stays free; inverting
the relationship would reinstate the mute arm that spoke on 2% of 11,768
calls.
"""
from scribe.services import plugin_context as pc
assert pc.TOOLRULE_DEFAULT_THRESHOLD < pc.RULEHINT_DEFAULT_THRESHOLD
def test_the_rule_bar_defaults_above_the_code_bar():
"""Not a number check — a DIRECTION check, and the only part of the default
that is defensible without one instance's histogram (rule 115).