feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
Milestone 333 step 4 — the split #2223 made one surface down, now made for the third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT. THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two facts hold on any install, including one with six rules and no telemetry: - The eligible corpus is tiny — conditional rules only, a handful to a few dozen against thousands of notes. A top-k over forty candidates always returns something, so "the best match cleared the bar" stops meaning "a good match exists". A bar calibrated for best-of-thousands is cleared by best-of-forty as arithmetic, not relevance. - Rules are short imperative technical English, far more homogeneous than note prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it; a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely inherited, it sits below where this corpus's noise lives. 0.72 errs deliberately toward silence on an asymmetry that is also structural: this hint fires on EVERY write. A missed rule is recoverable — it is still in Scribe and the agent can search it. A hint that cries wolf is not: it teaches the reader to skip the whole block, and the true positives go with it. The arm's own comment already said "noise on a hint that fires on every write is how a hint gets ignored". Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting the relationship — which would silently reinstate #3311 — does not. RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2 means the second line is almost always the second-best noise wearing the same confident framing as the first; halving k halves that regardless of the bar. It stays a constant because it is a decision about how loud one hint may be, not a per-install tuning question — and a knob nobody turns only adds a way to misconfigure the surface. Reachable from Settings, no restart (rule 25), with copy that says which way to move it and points at retrieval_telemetry's rule pull-through — which step 3 made readable — to tell "arriving unread" from "never arrived". Every config stand-in in the suite gained the key, not just the one that noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so a missing key raises inside its fail-open except and turns the arm into a silent no-op — indistinguishable from it running and finding nothing. That is the same vacuous-pass shape that bit step 2, one layer down (rule 33). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
@@ -47,17 +47,25 @@ _PRIOR_ART = [(0.72, fake_note(id=9, title="debounce helper", user_id=1,
|
||||
note_type="snippet"))]
|
||||
|
||||
|
||||
def _arm_patches(pc, hits, recorder, prior_art=None):
|
||||
"""The minimum stubbing that lets the rule arm run and nothing else."""
|
||||
def _arm_patches(pc, hits, recorder, prior_art=None, cfg=None, rule_search=None):
|
||||
"""The minimum stubbing that lets the rule arm run and nothing else.
|
||||
|
||||
`cfg` and `rule_search` are overridable so a caller can inspect what the
|
||||
arm ASKED for rather than only what it did with the answer — patching them
|
||||
a second time on top would work, but reads as an accident.
|
||||
"""
|
||||
return (
|
||||
patch.object(pc, "get_writepath_config",
|
||||
AsyncMock(return_value={"enabled": True, "threshold": 0.6,
|
||||
"top_k": 3})),
|
||||
AsyncMock(return_value=cfg or {
|
||||
"enabled": True, "threshold": 0.6,
|
||||
"top_k": 3, "rule_threshold": 0.6,
|
||||
})),
|
||||
patch.object(pc.snippets_svc, "list_snippets", AsyncMock(return_value=([], 0))),
|
||||
patch.object(pc, "semantic_search_notes",
|
||||
AsyncMock(return_value=_PRIOR_ART if prior_art is None
|
||||
else prior_art)),
|
||||
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
|
||||
patch.object(pc, "semantic_search_rules",
|
||||
rule_search or AsyncMock(return_value=hits)),
|
||||
patch.object(pc, "record_retrieval", MagicMock()),
|
||||
patch.object(pc, "record_surfaced", MagicMock()),
|
||||
patch.object(pc, "record_rule_surfaced", recorder),
|
||||
@@ -122,6 +130,36 @@ async def test_nothing_is_recorded_when_every_hit_was_already_held():
|
||||
assert rec.call_count == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
|
||||
"""The consuming half of step 4. `get_writepath_config` assembling a
|
||||
separate `rule_threshold` means nothing if the arm still passes
|
||||
`cfg["threshold"]` to its search — the split would exist in the config and
|
||||
not in the behaviour, and #3311 would be exactly where it was.
|
||||
|
||||
The two values are deliberately different here so the assertion can tell
|
||||
them apart.
|
||||
"""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
search = AsyncMock(return_value=[])
|
||||
with ExitStack() as stack:
|
||||
for ctx in _arm_patches(
|
||||
pc, [], MagicMock(), rule_search=search,
|
||||
cfg={"enabled": True, "threshold": 0.60,
|
||||
"top_k": 3, "rule_threshold": 0.81},
|
||||
):
|
||||
stack.enter_context(ctx)
|
||||
await pc.build_write_path_hint(
|
||||
1, "frontend/src/api/client.ts", code="x" * 400,
|
||||
)
|
||||
|
||||
kw = search.await_args.kwargs
|
||||
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
|
||||
assert kw["limit"] == pc.RULEHINT_LIMIT
|
||||
assert kw["tier"] == "conditional"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_arm_does_not_fire_on_a_write_that_matched_nothing():
|
||||
"""The gate, pinned — because the fixture above now depends on it and a
|
||||
|
||||
Reference in New Issue
Block a user