feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 35s
Milestone 333 step 4 — the split #2223 made one surface down, now made for the third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT. THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two facts hold on any install, including one with six rules and no telemetry: - The eligible corpus is tiny — conditional rules only, a handful to a few dozen against thousands of notes. A top-k over forty candidates always returns something, so "the best match cleared the bar" stops meaning "a good match exists". A bar calibrated for best-of-thousands is cleared by best-of-forty as arithmetic, not relevance. - Rules are short imperative technical English, far more homogeneous than note prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it; a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely inherited, it sits below where this corpus's noise lives. 0.72 errs deliberately toward silence on an asymmetry that is also structural: this hint fires on EVERY write. A missed rule is recoverable — it is still in Scribe and the agent can search it. A hint that cries wolf is not: it teaches the reader to skip the whole block, and the true positives go with it. The arm's own comment already said "noise on a hint that fires on every write is how a hint gets ignored". Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting the relationship — which would silently reinstate #3311 — does not. RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2 means the second line is almost always the second-best noise wearing the same confident framing as the first; halving k halves that regardless of the bar. It stays a constant because it is a decision about how loud one hint may be, not a per-install tuning question — and a knob nobody turns only adds a way to misconfigure the surface. Reachable from Settings, no restart (rule 25), with copy that says which way to move it and points at retrieval_telemetry's rule pull-through — which step 3 made readable — to tell "arriving unread" from "never arrived". Every config stand-in in the suite gained the key, not just the one that noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so a missing key raises inside its fail-open except and turns the arm into a silent no-op — indistinguishable from it running and finding nothing. That is the same vacuous-pass shape that bit step 2, one layer down (rule 33). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN
This commit is contained in:
@@ -86,6 +86,7 @@ const kbWritePathEnabled = ref(true);
|
||||
// code embeddings sit on a much higher similarity floor than prose, so 0.55 let
|
||||
// unrelated code through (#2223). Shares top-k, not the threshold.
|
||||
const kbWritePathThreshold = ref("0.68");
|
||||
const kbRuleHintThreshold = ref("0.72");
|
||||
// Near-duplicate report floors, one per record kind (services/dedup.py).
|
||||
// Snippets are single-chunk, so their floor sits below the 0.90 write-time
|
||||
// gate and catches what it lets through. Notes/tasks are scored at chunk
|
||||
@@ -148,12 +149,17 @@ async function saveKbInject() {
|
||||
// Same `|| default` reasoning: falling back to 0 would surface every
|
||||
// snippet in the corpus on every edit, which is the failure this knob fixes.
|
||||
const wpT = Math.min(1, Math.max(0, Number(kbWritePathThreshold.value) || 0.68));
|
||||
// Same `|| default` reasoning again, and it bites harder here: a rule hint
|
||||
// fires on every write, so a fallback of 0 would attach a standing rule to
|
||||
// every edit in the session.
|
||||
const rhT = Math.min(1, Math.max(0, Number(kbRuleHintThreshold.value) || 0.72));
|
||||
kbInjectThreshold.value = String(t);
|
||||
kbInjectTopK.value = String(k);
|
||||
kbDupThresholdSnippet.value = String(dupSnip);
|
||||
kbDupThresholdNote.value = String(dupNote);
|
||||
kbDupThresholdTask.value = String(dupTask);
|
||||
kbWritePathThreshold.value = String(wpT);
|
||||
kbRuleHintThreshold.value = String(rhT);
|
||||
savingKbInject.value = true;
|
||||
kbInjectSaved.value = false;
|
||||
try {
|
||||
@@ -166,6 +172,10 @@ async function saveKbInject() {
|
||||
// measurements that split them.
|
||||
kb_writepath_enabled: kbWritePathEnabled.value ? 'true' : 'false',
|
||||
kb_writepath_threshold: String(wpT),
|
||||
// A THIRD corpus with a third bar — see RULEHINT_DEFAULT_THRESHOLD
|
||||
// in services/plugin_context.py for why rules cannot share the
|
||||
// code threshold any more than code could share the prose one.
|
||||
kb_rulehint_threshold: String(rhT),
|
||||
kb_duplicate_threshold_snippet: String(dupSnip),
|
||||
kb_duplicate_threshold_note: String(dupNote),
|
||||
kb_duplicate_threshold_task: String(dupTask),
|
||||
@@ -611,6 +621,9 @@ onMounted(async () => {
|
||||
kbInjectTopK.value = allSettings.kb_autoinject_top_k;
|
||||
}
|
||||
kbWritePathEnabled.value = allSettings.kb_writepath_enabled !== "false";
|
||||
if (allSettings.kb_rulehint_threshold !== undefined) {
|
||||
kbRuleHintThreshold.value = allSettings.kb_rulehint_threshold;
|
||||
}
|
||||
if (allSettings.kb_writepath_threshold !== undefined) {
|
||||
kbWritePathThreshold.value = allSettings.kb_writepath_threshold;
|
||||
}
|
||||
@@ -1456,6 +1469,29 @@ async function deleteUser(userId: number) {
|
||||
location, not by resemblance.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-rulehint-threshold">Standing-rule confidence threshold (0–1)</label>
|
||||
<input
|
||||
id="kb-rulehint-threshold"
|
||||
v-model="kbRuleHintThreshold"
|
||||
type="number"
|
||||
min="0"
|
||||
max="1"
|
||||
step="0.01"
|
||||
class="fs-input input"
|
||||
style="max-width: 8rem"
|
||||
/>
|
||||
<p class="field-hint">
|
||||
The same hint can mention a standing rule whose trigger resembles what's
|
||||
being written — only rules marked <em>conditional</em>, since always-on
|
||||
ones are already loaded. Stricter again than the threshold above, because
|
||||
there are far fewer rules than snippets: with a small set, something
|
||||
always ranks first, so the bar has to carry more of the judgement.
|
||||
Raise it if rules keep arriving unread; lower it if a rule you needed
|
||||
never showed up. Settings → check the pull-through in
|
||||
<code>retrieval_telemetry</code> to see which is happening.
|
||||
</p>
|
||||
</div>
|
||||
<!-- A design system belongs to a PROJECT, and the picker for it lives on
|
||||
the project. There was a setting here that designated the system
|
||||
this install's own interface was built from; it only ever described
|
||||
|
||||
Reference in New Issue
Block a user