feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
Milestone 333 step 4 — the split #2223 made one surface down, now made for the third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT. THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two facts hold on any install, including one with six rules and no telemetry: - The eligible corpus is tiny — conditional rules only, a handful to a few dozen against thousands of notes. A top-k over forty candidates always returns something, so "the best match cleared the bar" stops meaning "a good match exists". A bar calibrated for best-of-thousands is cleared by best-of-forty as arithmetic, not relevance. - Rules are short imperative technical English, far more homogeneous than note prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it; a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely inherited, it sits below where this corpus's noise lives. 0.72 errs deliberately toward silence on an asymmetry that is also structural: this hint fires on EVERY write. A missed rule is recoverable — it is still in Scribe and the agent can search it. A hint that cries wolf is not: it teaches the reader to skip the whole block, and the true positives go with it. The arm's own comment already said "noise on a hint that fires on every write is how a hint gets ignored". Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting the relationship — which would silently reinstate #3311 — does not. RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2 means the second line is almost always the second-best noise wearing the same confident framing as the first; halving k halves that regardless of the bar. It stays a constant because it is a decision about how loud one hint may be, not a per-install tuning question — and a knob nobody turns only adds a way to misconfigure the surface. Reachable from Settings, no restart (rule 25), with copy that says which way to move it and points at retrieval_telemetry's rule pull-through — which step 3 made readable — to tell "arriving unread" from "never arrived". Every config stand-in in the suite gained the key, not just the one that noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so a missing key raises inside its fail-open except and turns the arm into a silent no-op — indistinguishable from it running and finding nothing. That is the same vacuous-pass shape that bit step 2, one layer down (rule 33). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
@@ -25,7 +25,13 @@ def _snippet_item(nid, title, user_id=1):
|
||||
|
||||
|
||||
def _cfg(**over):
|
||||
base = {"enabled": True, "threshold": 0.68, "top_k": 3}
|
||||
# `rule_threshold` is the standing-rule arm's own bar (milestone 333 step
|
||||
# 4). It belongs in the stand-in even though most tests here never reach
|
||||
# that arm: the arm reads it while BUILDING its search arguments, so a
|
||||
# missing key raises inside its fail-open except and turns the arm into a
|
||||
# silent no-op — which is indistinguishable from it working and finding
|
||||
# nothing.
|
||||
base = {"enabled": True, "threshold": 0.68, "top_k": 3, "rule_threshold": 0.72}
|
||||
base.update(over)
|
||||
return base
|
||||
|
||||
@@ -426,6 +432,74 @@ async def test_writepath_threshold_is_operator_tunable_and_clamped():
|
||||
assert (await _cfg_with("banana"))["threshold"] == pc.WRITEPATH_DEFAULT_THRESHOLD
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_rule_arm_has_its_own_tunable_bar():
|
||||
"""Rule #25 again, for the THIRD corpus (milestone 333 step 4).
|
||||
|
||||
Separate from the code threshold above and separately settable, because the
|
||||
two are measured against different things: 0.68 was derived from code
|
||||
against note PROSE (#2223), and rules are short imperative technical
|
||||
English — a more homogeneous corpus whose noise floor sits higher.
|
||||
"""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
async def _cfg_with(raw):
|
||||
stored = {pc.RULEHINT_THRESHOLD_KEY: raw}
|
||||
with patch.object(pc, "get_setting",
|
||||
AsyncMock(side_effect=lambda uid, k, d: stored.get(k, d))):
|
||||
return await pc.get_writepath_config(1)
|
||||
|
||||
assert (await _cfg_with("0.8"))["rule_threshold"] == 0.8
|
||||
assert (await _cfg_with("5"))["rule_threshold"] == 1.0
|
||||
assert (await _cfg_with("-3"))["rule_threshold"] == 0.0
|
||||
# Garbage falls back to the default, not to 0.0 — which on THIS arm would
|
||||
# attach a standing rule to every write in the session.
|
||||
assert (await _cfg_with("banana"))["rule_threshold"] == pc.RULEHINT_DEFAULT_THRESHOLD
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_two_write_path_bars_are_independent():
|
||||
"""The split, asserted. Setting one must not move the other — the failure
|
||||
that would silently undo this step is a config assembler that reads one key
|
||||
into both fields."""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
stored = {pc.WRITEPATH_THRESHOLD_KEY: "0.90", pc.RULEHINT_THRESHOLD_KEY: "0.61"}
|
||||
with patch.object(pc, "get_setting",
|
||||
AsyncMock(side_effect=lambda uid, k, d: stored.get(k, d))):
|
||||
cfg = await pc.get_writepath_config(1)
|
||||
|
||||
assert cfg["threshold"] == 0.90
|
||||
assert cfg["rule_threshold"] == 0.61
|
||||
|
||||
|
||||
def test_the_rule_bar_defaults_above_the_code_bar():
|
||||
"""Not a number check — a DIRECTION check, and the only part of the default
|
||||
that is defensible without one instance's histogram (rule 115).
|
||||
|
||||
The eligible rule corpus is orders of magnitude smaller than the note
|
||||
corpus, so a top-k over it always returns something and a bar calibrated
|
||||
for best-of-thousands is cleared by best-of-forty as arithmetic. Rules are
|
||||
also more homogeneous than note prose, so their noise floor is higher. Both
|
||||
facts point the same way: this bar must sit ABOVE the one it inherited.
|
||||
|
||||
Pinned as an inequality so tuning the value stays free while inverting the
|
||||
relationship — which would silently reinstate #3311 — does not.
|
||||
"""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
assert pc.RULEHINT_DEFAULT_THRESHOLD > pc.WRITEPATH_DEFAULT_THRESHOLD
|
||||
|
||||
|
||||
def test_the_rule_arm_asks_for_one_rule_not_two():
|
||||
"""With a corpus this small, top-k does as much damage as the threshold:
|
||||
k=2 over a few dozen candidates means the second line is almost always the
|
||||
second-best noise, carrying the same confident framing as the first."""
|
||||
from scribe.services import plugin_context as pc
|
||||
|
||||
assert pc.RULEHINT_LIMIT == 1
|
||||
|
||||
|
||||
# --- the minimum-substance floor on the semantic arm (#2223) ------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user