feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s

Milestone 333 step 4 — the split #2223 made one surface down, now made for the
third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number
measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT.

THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two
facts hold on any install, including one with six rules and no telemetry:

- The eligible corpus is tiny — conditional rules only, a handful to a few
  dozen against thousands of notes. A top-k over forty candidates always
  returns something, so "the best match cleared the bar" stops meaning "a good
  match exists". A bar calibrated for best-of-thousands is cleared by
  best-of-forty as arithmetic, not relevance.
- Rules are short imperative technical English, far more homogeneous than note
  prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it;
  a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely
  inherited, it sits below where this corpus's noise lives.

0.72 errs deliberately toward silence on an asymmetry that is also structural:
this hint fires on EVERY write. A missed rule is recoverable — it is still in
Scribe and the agent can search it. A hint that cries wolf is not: it teaches
the reader to skip the whole block, and the true positives go with it. The
arm's own comment already said "noise on a hint that fires on every write is
how a hint gets ignored".

Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar
asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting
the relationship — which would silently reinstate #3311 — does not.

RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2
means the second line is almost always the second-best noise wearing the same
confident framing as the first; halving k halves that regardless of the bar.
It stays a constant because it is a decision about how loud one hint may be,
not a per-install tuning question — and a knob nobody turns only adds a way to
misconfigure the surface.

Reachable from Settings, no restart (rule 25), with copy that says which way to
move it and points at retrieval_telemetry's rule pull-through — which step 3
made readable — to tell "arriving unread" from "never arrived".

Every config stand-in in the suite gained the key, not just the one that
noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so
a missing key raises inside its fail-open except and turns the arm into a
silent no-op — indistinguishable from it running and finding nothing. That is
the same vacuous-pass shape that bit step 2, one layer down (rule 33).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
2026-09-02 18:05:00 -04:00
co-authored by Claude Opus 5
parent 8901c904a9
commit 238510080e
6 changed files with 227 additions and 12 deletions
+43 -5
View File
@@ -47,17 +47,25 @@ _PRIOR_ART = [(0.72, fake_note(id=9, title="debounce helper", user_id=1,
note_type="snippet"))]
def _arm_patches(pc, hits, recorder, prior_art=None):
"""The minimum stubbing that lets the rule arm run and nothing else."""
def _arm_patches(pc, hits, recorder, prior_art=None, cfg=None, rule_search=None):
"""The minimum stubbing that lets the rule arm run and nothing else.
`cfg` and `rule_search` are overridable so a caller can inspect what the
arm ASKED for rather than only what it did with the answer — patching them
a second time on top would work, but reads as an accident.
"""
return (
patch.object(pc, "get_writepath_config",
AsyncMock(return_value={"enabled": True, "threshold": 0.6,
"top_k": 3})),
AsyncMock(return_value=cfg or {
"enabled": True, "threshold": 0.6,
"top_k": 3, "rule_threshold": 0.6,
})),
patch.object(pc.snippets_svc, "list_snippets", AsyncMock(return_value=([], 0))),
patch.object(pc, "semantic_search_notes",
AsyncMock(return_value=_PRIOR_ART if prior_art is None
else prior_art)),
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
patch.object(pc, "semantic_search_rules",
rule_search or AsyncMock(return_value=hits)),
patch.object(pc, "record_retrieval", MagicMock()),
patch.object(pc, "record_surfaced", MagicMock()),
patch.object(pc, "record_rule_surfaced", recorder),
@@ -122,6 +130,36 @@ async def test_nothing_is_recorded_when_every_hit_was_already_held():
assert rec.call_count == 0
@pytest.mark.asyncio
async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
"""The consuming half of step 4. `get_writepath_config` assembling a
separate `rule_threshold` means nothing if the arm still passes
`cfg["threshold"]` to its search — the split would exist in the config and
not in the behaviour, and #3311 would be exactly where it was.
The two values are deliberately different here so the assertion can tell
them apart.
"""
from scribe.services import plugin_context as pc
search = AsyncMock(return_value=[])
with ExitStack() as stack:
for ctx in _arm_patches(
pc, [], MagicMock(), rule_search=search,
cfg={"enabled": True, "threshold": 0.60,
"top_k": 3, "rule_threshold": 0.81},
):
stack.enter_context(ctx)
await pc.build_write_path_hint(
1, "frontend/src/api/client.ts", code="x" * 400,
)
kw = search.await_args.kwargs
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
assert kw["limit"] == pc.RULEHINT_LIMIT
assert kw["tier"] == "conditional"
@pytest.mark.asyncio
async def test_the_arm_does_not_fire_on_a_write_that_matched_nothing():
"""The gate, pinned — because the fixture above now depends on it and a