align(4909d): every rule arm that annotates reads one bar, kb_rule_threshold at 0.70
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped

prompt_rule, pre_tool_rule and write_path_rule now share one settings key,
RULE_FLOOR_KEY = "kb_rule_threshold", default 0.70. The reply backstop keeps
its own 0.80, because there the floor is a stop bar. Moving the shared floor
through tune_retrieval or the Settings form records a tuning event on every
surface that reads it (set_dial, via retrieval_surfaces.sharing_floor), and
the result names the surfaces it carried as `moved_with`. Settings shows one
"Standing-rule confidence threshold" control where it showed three.

Operator ruling (2026-10-05): one shared bar for the rule arms, "especially
since we're only taking the first x number of them" -- the budget limits
what is shown; the bar only decides what is worth ranking. Kept a dial.

Why 0.70, from records read rather than percentiles:
- write_path_rule near-misses at 0.7199-0.72 were all false positives
  (rule 149 against a Lidarr comment, 157 against a retention Svelte file,
  121 against an Archives restore, 164 against a config test, 131 against a
  Go package), so the old 0.72 was not hiding real hits -- but the band does
  noise control on this arm, so dropping 0.02 costs little.
- prompt_rule: the operator set 0.70 by hand on 2026-09-18, after a 0.60
  experiment admitted junk and was reverted. 0.70 is the number with a human
  decision behind it.
- pre_tool_rule lines near 0.68 this session were mostly noise; rising to
  0.70 trims them, and the band still narrows what clears.

Retired keys: kb_promptrule_threshold, kb_toolrule_threshold and
kb_rulehint_threshold are no longer read. Rows an install stored under them
stay in the settings table, inert. On this instance the effective bars move
pre_tool_rule 0.68 -> 0.70, write_path_rule 0.72 -> 0.70, prompt_rule stays
0.70; record_release_defaults writes those per-surface release rows on boot,
so the telemetry shift has an event to explain it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-22 18:32:09 -04:00
co-authored by Claude Opus 5.5
parent f77d18cb63
commit c8ede427ca
10 changed files with 175 additions and 211 deletions
+25 -88
View File
@@ -159,16 +159,10 @@ const kbWritePathEnabled = ref(true);
// code embeddings sit on a much higher similarity floor than prose, so 0.55 // code embeddings sit on a much higher similarity floor than prose, so 0.55
// lets unrelated code through. Shares top-k, not the threshold. // lets unrelated code through. Shares top-k, not the threshold.
const kbWritePathThreshold = ref("0.68"); const kbWritePathThreshold = ref("0.68");
const kbRuleHintThreshold = ref("0.72"); // One bar for every rule hint — prompt, command and edit alike. They search
// the same rules, and what limits each is its budget, not the bar.
const kbRuleThreshold = ref("0.70");
const kbCheckpointThreshold = ref("0.80"); const kbCheckpointThreshold = ref("0.80");
// The two ACT arms do not share a bar. A write-path query is a code payload; a
// pre-tool query is a shell command, often under a dozen words — less text,
// less signal, lower scores for the same relevance — so one shared bar makes
// them behave like different subsystems.
const kbToolRuleThreshold = ref("0.68");
// And the prompt boundary is a third query shape again — the operator's own
// prose rather than anything a tool produced.
const kbPromptRuleThreshold = ref("0.72");
// The reply arm: its bar is the bar at which a finished reply // The reply arm: its bar is the bar at which a finished reply
// is HELD for one read, so it sits with the checkpoint, not with the hints. // is HELD for one read, so it sits with the checkpoint, not with the hints.
const kbReplyRuleThreshold = ref("0.8"); const kbReplyRuleThreshold = ref("0.8");
@@ -367,14 +361,10 @@ async function saveKbInject() {
// Same `|| default` reasoning: falling back to 0 would surface every // Same `|| default` reasoning: falling back to 0 would surface every
// snippet in the corpus on every edit, which is the failure this knob fixes. // snippet in the corpus on every edit, which is the failure this knob fixes.
const wpT = asBar(kbWritePathThreshold.value, 0.68); const wpT = asBar(kbWritePathThreshold.value, 0.68);
// Same `|| default` reasoning again, and it bites harder here: a rule hint // Same `|| default` reasoning again, and it bites harder here: rule hints
// fires on every write, so a fallback of 0 would attach a standing rule to // fire on every edit and before every command, so a fallback of 0 would put
// every edit in the session. // a standing rule beside each of them.
const rhT = asBar(kbRuleHintThreshold.value, 0.72); const ruleT = asBar(kbRuleThreshold.value, 0.7);
// Same `|| default` guard, and the same reason: this arm fires before every
// Bash call, so a fallback of 0 would put a rule in front of every command.
const trT = asBar(kbToolRuleThreshold.value, 0.68);
const prT = asBar(kbPromptRuleThreshold.value, 0.72);
// The checkpoint bar, and the `|| default` guard matters most here of all: // The checkpoint bar, and the `|| default` guard matters most here of all:
// this is the only number that can STOP a call, so a fallback of 0 would // this is the only number that can STOP a call, so a fallback of 0 would
// hold the first command of every session behind whatever ranked first. // hold the first command of every session behind whatever ranked first.
@@ -409,10 +399,8 @@ async function saveKbInject() {
kbGateNoteOverlapFloor.value = String(gateOverlap); kbGateNoteOverlapFloor.value = String(gateOverlap);
kbPlanMatchThreshold.value = String(planT); kbPlanMatchThreshold.value = String(planT);
kbWritePathThreshold.value = String(wpT); kbWritePathThreshold.value = String(wpT);
kbRuleHintThreshold.value = String(rhT); kbRuleThreshold.value = String(ruleT);
kbCheckpointThreshold.value = String(cpT); kbCheckpointThreshold.value = String(cpT);
kbToolRuleThreshold.value = String(trT);
kbPromptRuleThreshold.value = String(prT);
kbReplyRuleThreshold.value = String(ryT); kbReplyRuleThreshold.value = String(ryT);
savingKbInject.value = true; savingKbInject.value = true;
kbInjectSaved.value = false; kbInjectSaved.value = false;
@@ -426,20 +414,15 @@ async function saveKbInject() {
// measurements that split them. // measurements that split them.
kb_writepath_enabled: kbWritePathEnabled.value ? 'true' : 'false', kb_writepath_enabled: kbWritePathEnabled.value ? 'true' : 'false',
kb_writepath_threshold: String(wpT), kb_writepath_threshold: String(wpT),
// A THIRD corpus with a third bar — see RULEHINT_DEFAULT_THRESHOLD // A third corpus with a third bar — see RULE_FLOOR_DEFAULT in
// in services/plugin_context.py for why rules cannot share the // services/retrieval_pipeline.py for why every rule hint shares it.
// code threshold any more than code could share the prose one. kb_rule_threshold: String(ruleT),
kb_rulehint_threshold: String(rhT),
// The one bar that stops a call rather than annotating it. Its own key, // The one bar that stops a call rather than annotating it. Its own key,
// never derived from the two rule bars above: it answers a different // never derived from the rule bar above: it answers a different
// question of the same scores — not "is this worth showing" but "is the // question of the same scores — not "is this worth showing" but "is the
// corpus confident enough to be read first" — and a number that moves // corpus confident enough to be read first" — and a number that moves
// when another moves is a number nobody can reason about. // when another moves is a number nobody can reason about.
kb_checkpoint_threshold: String(cpT), kb_checkpoint_threshold: String(cpT),
// A FOURTH and FIFTH bar, and they are separate keys on purpose: one
// number cannot serve arms whose queries are different shapes. Moving one must not move the others.
kb_toolrule_threshold: String(trT),
kb_promptrule_threshold: String(prT),
// The reply backstop's own bar: it HOLDS a reply, so like the // The reply backstop's own bar: it HOLDS a reply, so like the
// checkpoint it is never derived from a hint bar. // checkpoint it is never derived from a hint bar.
kb_replyrule_threshold: String(ryT), kb_replyrule_threshold: String(ryT),
@@ -864,14 +847,8 @@ onMounted(async () => {
if (allSettings.kb_checkpoint_threshold !== undefined) { if (allSettings.kb_checkpoint_threshold !== undefined) {
kbCheckpointThreshold.value = allSettings.kb_checkpoint_threshold; kbCheckpointThreshold.value = allSettings.kb_checkpoint_threshold;
} }
if (allSettings.kb_rulehint_threshold !== undefined) { if (allSettings.kb_rule_threshold !== undefined) {
kbRuleHintThreshold.value = allSettings.kb_rulehint_threshold; kbRuleThreshold.value = allSettings.kb_rule_threshold;
}
if (allSettings.kb_toolrule_threshold !== undefined) {
kbToolRuleThreshold.value = allSettings.kb_toolrule_threshold;
}
if (allSettings.kb_promptrule_threshold !== undefined) {
kbPromptRuleThreshold.value = allSettings.kb_promptrule_threshold;
} }
if (allSettings.kb_replyrule_threshold !== undefined) { if (allSettings.kb_replyrule_threshold !== undefined) {
kbReplyRuleThreshold.value = allSettings.kb_replyrule_threshold; kbReplyRuleThreshold.value = allSettings.kb_replyrule_threshold;
@@ -1772,10 +1749,10 @@ async function deleteUser(userId: number) {
<p class="field-hint">How many snippets and issues one edit may be offered (1–10). Reach for this rather than the threshold when the hint feels long: lowering it keeps the best matches and drops the tail, while raising the bar above throws away good matches along with weak ones.</p> <p class="field-hint">How many snippets and issues one edit may be offered (1–10). Reach for this rather than the threshold when the hint feels long: lowering it keeps the best matches and drops the tail, while raising the bar above throws away good matches along with weak ones.</p>
</div> </div>
<div class="field"> <div class="field">
<label for="kb-rulehint-threshold">Standing-rule confidence threshold (0–1)</label> <label for="kb-rule-threshold">Standing-rule confidence threshold (0–1)</label>
<input <input
id="kb-rulehint-threshold" id="kb-rule-threshold"
v-model="kbRuleHintThreshold" v-model="kbRuleThreshold"
type="number" type="number"
min="0" min="0"
max="1" max="1"
@@ -1784,14 +1761,14 @@ async function deleteUser(userId: number) {
style="max-width: 8rem" style="max-width: 8rem"
/> />
<p class="field-hint"> <p class="field-hint">
The same hint can mention a standing rule whose trigger resembles the The bar for a standing rule or preference to be offered at all,
code being written. <em>Every</em> rule is eligible — nothing is whether against the code being written, a command about to run, or
preloaded any more, so this is the only way a rule reaches a write. what you just said. <em>Every</em> rule is eligible — nothing is
Stricter than the threshold above, because there are far fewer rules preloaded, so this is how a rule reaches the work. One bar serves all
than snippets: with a small set something always ranks first, so the three: they search the same rules, and how many each may show is set
bar has to carry more of the judgement. If rules arrive unread, or one by its own budget below, which is usually the better lever. If rules
you needed never showed up, that is Claude's to notice and correct arrive unread, or one you needed never showed up, that is Claude's to
from the telemetry — and the budget below is usually the better lever. notice and correct from the telemetry.
</p> </p>
</div> </div>
<div class="field"> <div class="field">
@@ -1834,26 +1811,6 @@ async function deleteUser(userId: number) {
only — file edits are never held. only — file edits are never held.
</p> </p>
</div> </div>
<div class="field">
<label for="kb-toolrule-threshold">Command confidence threshold (0–1)</label>
<input
id="kb-toolrule-threshold"
v-model="kbToolRuleThreshold"
type="number"
min="0"
max="1"
step="0.01"
class="fs-input input"
style="max-width: 8rem"
/>
<p class="field-hint">
The bar for a rule surfacing before a <em>command</em> runs, where the
query is the command text rather than code. Lower than the one above
on purpose: a shell command is short, so it scores lower for the same
relevance — at a shared bar this arm spoke on 2% of calls against the
write path's 37%.
</p>
</div>
<div class="field"> <div class="field">
<label for="kb-toolrule-topk">Max rules per command</label> <label for="kb-toolrule-topk">Max rules per command</label>
<input <input
@@ -1868,26 +1825,6 @@ async function deleteUser(userId: number) {
/> />
<p class="field-hint">How many standing rules one command may be shown (1–10). This is the busiest arm there is — it fires before every command — so its budget is the one most worth keeping small.</p> <p class="field-hint">How many standing rules one command may be shown (1–10). This is the busiest arm there is — it fires before every command — so its budget is the one most worth keeping small.</p>
</div> </div>
<div class="field">
<label for="kb-promptrule-threshold">Prompt confidence threshold (0–1)</label>
<input
id="kb-promptrule-threshold"
v-model="kbPromptRuleThreshold"
type="number"
min="0"
max="1"
step="0.01"
class="fs-input input"
style="max-width: 8rem"
/>
<p class="field-hint">
The bar for a rule or preference surfacing against <em>what you just
said</em>, before anything is done. It is the only moment that reaches
a rule about how to answer rather than how to act, so it runs once a
turn rather than once a tool call. A third query shape — your prose,
not a command or a file — which is why it carries its own number.
</p>
</div>
<div class="field"> <div class="field">
<label for="kb-promptrule-topk">Max rules per prompt</label> <label for="kb-promptrule-topk">Max rules per prompt</label>
<input <input
+20 -64
View File
@@ -144,7 +144,7 @@ WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
WRITEPATH_DEFAULT_ENABLED = True WRITEPATH_DEFAULT_ENABLED = True
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
# The standing-rule arm gets its own bar, for a third corpus: 0.68 above is # The standing-rule arms get their own bar, for a third corpus: 0.68 above is
# calibrated on code against note PROSE, not code against RULE TEXT. # calibrated on code against note PROSE, not code against RULE TEXT.
# #
# THE STRUCTURAL ARGUMENT, which holds on any install, including one with six # THE STRUCTURAL ARGUMENT, which holds on any install, including one with six
@@ -155,64 +155,27 @@ WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
# pool always returns something, so "the best match cleared the bar" # pool always returns something, so "the best match cleared the bar"
# drifts from "a good match exists" toward "N things were ranked". A bar # drifts from "a good match exists" toward "N things were ranked". A bar
# calibrated for best-of-thousands is cleared by best-of-forty as # calibrated for best-of-thousands is cleared by best-of-forty as
# arithmetic rather than relevance. (The arms search every tier, which # arithmetic rather than relevance.
# makes the pool larger; the bar still errs toward silence on purpose.)
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS # 2. Rules are short imperative technical English — a far more HOMOGENEOUS
# corpus than note prose, and a more homogeneous corpus has a HIGHER # corpus than note prose, and a more homogeneous corpus has a HIGHER
# noise floor than prose's 0.55-0.63. # noise floor than prose's 0.55-0.63.
# #
# WHY 0.72 AND NOT A NUMBER OFF A HISTOGRAM. The exact offset between prose's # ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The write path, the command arm
# floor and rule-text's is not derivable in general — it depends on how an # and the prompt arm read the same key (`RULE_FLOOR_KEY` in retrieval_pipeline):
# install writes its rules — so the default errs deliberately toward SILENCE # they search the same rules, and what limits each one is its budget and band.
# rather than toward recall, on an asymmetry that is itself structural: this
# hint fires on EVERY write. A missed rule is recoverable, because the rule is
# still in Grimoire and the agent can search it. A hint that cries wolf is not:
# it teaches the reader to skip the whole block, and the surface is lost along
# with the true positives it would have carried.
# #
# TUNE IT FROM YOUR OWN INSTANCE: `retrieval_telemetry` reports # TUNE IT FROM YOUR OWN INSTANCE: `retrieval_telemetry` reports
# `rule_usage.pull_through`. Raise this if rules arrive unread; lower it if # `rule_usage.pull_through` and `near_miss_samples`. Raise it if rules arrive
# rules you needed never arrived. What would RETIRE it: a cross-encoder rerank, # unread; lower it if rules you needed never arrived. What would RETIRE it: a
# which would make a similarity bar the wrong control entirely. # cross-encoder rerank, which would make a similarity bar the wrong control
# # entirely.
# This bar is the WRITE-PATH arm's. A code payload is long and rich, which is
# the case 0.72 suits; the command arm has its own bar below.
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S. # THE COMMAND ARM READS THE SAME BAR. A shell command is short, so it scores
# # lower than a code payload for the same relevance and the shared bar admits
# A write-path query is a code payload, long and rich, while a pre-tool query # less here. The band does noise control downstream: a hit that clears the bar
# is a shell command — often under a dozen words. Less text, less signal, lower # still has to score within `_RULEHINT_BAND` of the top to be rendered.
# scores for the same relevance. At the write path's 0.72 the command arm
# turns nearly everything away, with its refused mass piled just under the
# line, which is the shape a bar set too high leaves.
#
# WHAT 0.68 IS CALIBRATED AGAINST. Replayed queries, consequential acts
# against innocuous ones, scored against a rule corpus with symptom-bearing
# triggers:
#
# 0.7571 git push origin dev consequential
# 0.7245 cd ...; git fetch; git add -A consequential
# 0.7193 git pull --rebase origin dev consequential
# 0.6850 docker compose up -d consequential
# ---------------------------------------- 0.68
# 0.6735 wc -l src/*.py && date innocuous
# 0.6544 grep -rn useState src/ innocuous
# 0.6099 sed -n '120,160p' package.json innocuous
# 0.6056 ls -la && cat README.md innocuous
#
# At 0.72 three of the four consequential acts would retrieve nothing, though
# the right rules ranked first for each.
#
# THE SEPARATION IS NARROW (about 0.01), and that is a caveat, not a result.
# Eight probes set a direction; they do not settle a number.
# `near_miss_samples` on real traffic is what settles it, and this is the bar
# to re-read first.
#
# WHY A LOWER BAR IS SAFE HERE. The band does noise control downstream: a marginal hit that
# clears the bar still has to score within `_RULEHINT_BAND` of the top to be
# rendered. The bar's job shrank, so the bar can.
# #
# The innocuous scores near the line are largely CROSS-PROJECT BLEED rather # The innocuous scores near the line are largely CROSS-PROJECT BLEED rather
# than bad ranking: another project's shell-command rule matching a shell # than bad ranking: another project's shell-command rule matching a shell
@@ -478,17 +441,11 @@ def _autoinject_query(prompt: str, context: str) -> str:
# The operator's message is the only query that exists before one is composed, # The operator's message is the only query that exists before one is composed,
# and this arm is what runs against it. # and this arm is what runs against it.
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
# INHERITED FROM THE ACT ARMS. 0.72 is tuned against # THE SAME BAR AS THE ACT ARMS. An operator's prose is a different query shape
# code and shell commands. An operator's prose is a different query shape # against the same documents, and triggers are written in the vocabulary of
# against the same documents, and nothing yet says the two distributions line # the moment, which for most rules is act vocabulary. Every call is logged
# up — triggers are written in the vocabulary of the MOMENT, which for most # under `prompt_rule`, so read `near_miss_samples` before moving the shared
# rules is act vocabulary, so prose may well score lower across the board. # bar, not the percentile alone.
#
# Starting at the act arms' number anyway is deliberate: it is the only value
# with evidence behind it, and guessing lower would put an unmeasured bar in
# front of a corpus that binds. Every call is logged under `prompt_rule`, so
# real traffic settles it — read `near_miss_samples` before moving this, not
# the percentile alone.
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
# ANCHORED ON THIS HOOK'S BUDGET rather than the act arms'. This arm fires # ANCHORED ON THIS HOOK'S BUDGET rather than the act arms'. This arm fires
@@ -1171,9 +1128,8 @@ async def get_writepath_config(user_id: int) -> dict:
independent pairs, resolved from the registry. independent pairs, resolved from the registry.
Each floor's reason is recorded where its default is: WRITEPATH (code Each floor's reason is recorded where its default is: WRITEPATH (code
embeddings sit on a much higher similarity floor than prose), RULEHINT (a embeddings sit on a much higher similarity floor than prose) and RULEHINT
third corpus again), TOOLRULE (a shell command is a different query shape (a third corpus again, whose one bar both rule arms read).
from a code payload and scores lower for the same relevance).
THE BUDGET IS SPLIT TOO. "How many titles at once" does not mean the same THE BUDGET IS SPLIT TOO. "How many titles at once" does not mean the same
thing on both surfaces: this arm fires before every Write and Edit while thing on both surfaces: this arm fires before every Write and Edit while
+20 -6
View File
@@ -460,6 +460,20 @@ class RuleArm:
declared: Declared | None = None declared: Declared | None = None
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The prompt, command and write
# arms search the same corpus, and what limits what they show is the budget
# and the band, not the floor. The floor only answers "is this worth ranking
# at all", and that question has one answer per corpus, not one per query
# shape — so the three share one settings key, and moving it moves all three.
# `set_dial` records the move on every surface that shares the key.
#
# Set low on purpose: a hit under it is never shown, so a bar set too high
# costs a rule that binds, while one set a little low costs a line the band
# usually drops. The reply backstop keeps its own number, because there the
# floor is a stop bar and not a hint bar.
RULE_FLOOR_KEY = "kb_rule_threshold"
RULE_FLOOR_DEFAULT = 0.70
# The arms differ on purpose where these flags differ. Whether the prompt arm # The arms differ on purpose where these flags differ. Whether the prompt arm
# should band, and whether the act arms should reserve a preference, are open # should band, and whether the act arms should reserve a preference, are open
# questions to settle on telemetry, not by symmetry. # questions to settle on telemetry, not by symmetry.
@@ -468,8 +482,8 @@ PROMPT_RULE = RuleArm(
preference_slot=True, preference_slot=True,
tuning=Surface( tuning=Surface(
name="prompt_rule", name="prompt_rule",
floor_key="kb_promptrule_threshold", floor_key=RULE_FLOOR_KEY,
floor_default=0.72, floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_promptrule_top_k", budget_key="kb_promptrule_top_k",
budget_default=3, budget_default=3,
asks="the operator's message, against rule triggers", asks="the operator's message, against rule triggers",
@@ -483,8 +497,8 @@ PRE_TOOL_RULE = RuleArm(
preference_slot=False, preference_slot=False,
tuning=Surface( tuning=Surface(
name="pre_tool_rule", name="pre_tool_rule",
floor_key="kb_toolrule_threshold", floor_key=RULE_FLOOR_KEY,
floor_default=0.68, floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_toolrule_top_k", budget_key="kb_toolrule_top_k",
budget_default=5, budget_default=5,
asks="the command about to run, against rule triggers", asks="the command about to run, against rule triggers",
@@ -498,8 +512,8 @@ WRITE_PATH_RULE = RuleArm(
preference_slot=False, preference_slot=False,
tuning=Surface( tuning=Surface(
name="write_path_rule", name="write_path_rule",
floor_key="kb_rulehint_threshold", floor_key=RULE_FLOOR_KEY,
floor_default=0.72, floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_rulehint_top_k", budget_key="kb_rulehint_top_k",
budget_default=5, budget_default=5,
asks="the code being written, against rule triggers", asks="the code being written, against rule triggers",
@@ -110,6 +110,9 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
Derived rather than listed so a seventh surface is covered the moment it is Derived rather than listed so a seventh surface is covered the moment it is
added here, which is the only way this stays true. added here, which is the only way this stays true.
A floor key several surfaces share resolves to the first of them;
`set_dial` records the move on every surface in `sharing_floor`.
""" """
for surface in SURFACES.values(): for surface in SURFACES.values():
if key == surface.floor_key: if key == surface.floor_key:
@@ -119,6 +122,16 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
return None return None
def sharing_floor(name: str) -> list[str]:
"""Every surface that reads the same floor key as `name`, itself included.
The rule arms share one bar, so moving one surface's floor moves its
siblings, and each of them needs the event that says so.
"""
key = get_surface(name).floor_key
return [n for n, s in SURFACES.items() if s.floor_key == key]
async def floor_for(user_id: int, name: str) -> float: async def floor_for(user_id: int, name: str) -> float:
"""This install's current floor for a surface, clamped to [0, 1].""" """This install's current floor for a surface, clamped to [0, 1]."""
s = get_surface(name) s = get_surface(name)
+15 -6
View File
@@ -44,6 +44,7 @@ from grimoire.services.retrieval_surfaces import (
budget_for, budget_for,
floor_for, floor_for,
get_surface, get_surface,
sharing_floor,
surface_names, surface_names,
) )
from grimoire.services.settings import set_setting from grimoire.services.settings import set_setting
@@ -257,6 +258,10 @@ async def set_dial(
"reason": text, "actor": actor, "unchanged": True, "reason": text, "actor": actor, "unchanged": True,
} }
# A shared floor moves every surface that reads it, and each one's history
# has to show the move, or its telemetry changes with no event to explain it.
moved = sharing_floor(surface) if dial == "floor" else [surface]
await set_setting(user_id, key, stored) await set_setting(user_id, key, stored)
async with async_session() as session: async with async_session() as session:
# Stamped with the space this number was chosen in. Read at # Stamped with the space this number was chosen in. Read at
@@ -264,12 +269,13 @@ async def set_dial(
# the caller recording it are the same call, so there is no window in # the caller recording it are the same call, so there is no window in
# which they could disagree. # which they could disagree.
stamp = calibration_stamp() stamp = calibration_stamp()
session.add(RetrievalTuningEvent( for name in moved:
user_id=user_id, surface=surface, dial=dial, session.add(RetrievalTuningEvent(
old_value=old, new_value=applied, actor=actor, reason=text, user_id=user_id, surface=name, dial=dial,
embedding_model=stamp["embedding_model"], old_value=old, new_value=applied, actor=actor, reason=text,
shape_version=stamp["shape_version"], embedding_model=stamp["embedding_model"],
)) shape_version=stamp["shape_version"],
))
await session.commit() await session.commit()
return { return {
@@ -286,6 +292,9 @@ async def set_dial(
# Always present, both ways round: a caller that has to test for the # Always present, both ways round: a caller that has to test for the
# key's absence to learn the answer will eventually forget to. # key's absence to learn the answer will eventually forget to.
"unchanged": False, "unchanged": False,
# The other surfaces this move carried with it: a shared floor is one
# number, and a caller tuning one rule arm has tuned them all.
"moved_with": [n for n in moved if n != surface],
} }
+21 -5
View File
@@ -71,13 +71,29 @@ def test_every_surface_name_is_a_real_telemetry_source():
) )
def test_no_two_surfaces_share_a_settings_key(): def test_a_shared_settings_key_is_one_bar_with_one_default():
"""One dial, one label.""" """One dial, one label. Surfaces may share a floor key, as the rule arms
floors = [s.floor_key for s in rs.SURFACES.values()] do, but then they must ship the same default: otherwise the one number
means something different on each surface until somebody sets it."""
by_key: dict[str, set[float]] = {}
for s in rs.SURFACES.values():
by_key.setdefault(s.floor_key, set()).add(s.floor_default)
split = {k: v for k, v in by_key.items() if len(v) > 1}
assert not split, f"a shared floor key ships two defaults: {split}"
budgets = [s.budget_key for s in rs.SURFACES.values()] budgets = [s.budget_key for s in rs.SURFACES.values()]
assert len(set(floors)) == len(floors), f"duplicate floor key in {floors}"
assert len(set(budgets)) == len(budgets), f"duplicate budget key in {budgets}" assert len(set(budgets)) == len(budgets), f"duplicate budget key in {budgets}"
assert not (set(floors) & set(budgets)), "a floor key doubles as a budget key" assert not (set(by_key) & set(budgets)), "a floor key doubles as a budget key"
def test_the_rule_arms_that_annotate_share_one_bar_and_the_reply_stop_does_not():
from grimoire.services.retrieval_pipeline import RULE_FLOOR_KEY
assert sorted(rs.sharing_floor("prompt_rule")) == [
"pre_tool_rule", "prompt_rule", "write_path_rule",
]
assert rs.SURFACES["prompt_rule"].floor_key == RULE_FLOOR_KEY
assert rs.sharing_floor("reply_rule") == ["reply_rule"]
def test_every_surface_says_what_it_asks_over_what_and_how_often(): def test_every_surface_says_what_it_asks_over_what_and_how_often():
+33 -2
View File
@@ -86,9 +86,9 @@ async def test_a_good_change_writes_the_setting_and_the_event():
setter.assert_awaited_once() setter.assert_awaited_once()
_uid, key, stored = setter.await_args.args _uid, key, stored = setter.await_args.args
assert key == "kb_promptrule_threshold" and stored == "0.66" assert key == "kb_rule_threshold" and stored == "0.66"
event = session.add.call_args.args[0] event = session.add.call_args_list[0].args[0]
assert event.surface == "prompt_rule" and event.dial == "floor" assert event.surface == "prompt_rule" and event.dial == "floor"
# The before-value is what makes the history answerable. # The before-value is what makes the history answerable.
assert event.old_value == 0.72 and event.new_value == 0.66 assert event.old_value == 0.72 and event.new_value == 0.66
@@ -97,6 +97,37 @@ async def test_a_good_change_writes_the_setting_and_the_event():
assert out["clamped"] is False assert out["clamped"] is False
@pytest.mark.asyncio
async def test_a_shared_floor_records_the_move_on_every_surface_that_reads_it():
"""One key, one write, an event per surface. A rule arm whose bar moved
with no event of its own would show a telemetry shift nothing explains."""
session, ctx = _patches(floor=0.70)
with ctx[0], patch.object(rt, "set_setting", AsyncMock()) as setter, \
ctx[2], ctx[3]:
out = await rt.set_dial(
1, "pre_tool_rule", "floor", 0.74,
reason="read the 10 highest declines: every one was a real miss",
)
setter.assert_awaited_once()
recorded = {c.args[0].surface for c in session.add.call_args_list}
assert recorded == {"pre_tool_rule", "prompt_rule", "write_path_rule"}
assert sorted(out["moved_with"]) == ["prompt_rule", "write_path_rule"]
@pytest.mark.asyncio
async def test_a_budget_moves_only_its_own_surface():
session, ctx = _patches(budget=3)
with ctx[0], patch.object(rt, "set_setting", AsyncMock()), ctx[2], ctx[3]:
out = await rt.set_dial(
1, "prompt_rule", "budget", 4,
reason="on-point hits sit at ranks 4 and 5 as often as rank 1",
)
assert [c.args[0].surface for c in session.add.call_args_list] == ["prompt_rule"]
assert out["moved_with"] == []
@pytest.mark.asyncio @pytest.mark.asyncio
@pytest.mark.parametrize("dial, sent, applied", [ @pytest.mark.parametrize("dial, sent, applied", [
("floor", 1.4, 1.0), ("floor", 1.4, 1.0),
+7 -2
View File
@@ -76,10 +76,15 @@ def test_the_route_does_not_take_the_actor_from_the_caller():
def test_every_registry_key_is_recognised_as_a_dial(): def test_every_registry_key_is_recognised_as_a_dial():
"""Derived from the registry, both directions. A seventh surface added """Derived from the registry, both directions. A seventh surface added
without a row here is a floor that can be written silently again.""" without a row here is a floor that can be written silently again."""
from grimoire.services.retrieval_surfaces import SURFACES, dial_for_key from grimoire.services.retrieval_surfaces import (
SURFACES, dial_for_key, sharing_floor,
)
for name, surface in SURFACES.items(): for name, surface in SURFACES.items():
assert dial_for_key(surface.floor_key) == (name, "floor") # A shared floor resolves to one of the surfaces that read it, and
# `set_dial` records the move on all of them.
owner, dial = dial_for_key(surface.floor_key)
assert dial == "floor" and name in sharing_floor(owner)
assert dial_for_key(surface.budget_key) == (name, "budget") assert dial_for_key(surface.budget_key) == (name, "budget")
+5 -4
View File
@@ -41,7 +41,7 @@ ROOT = pathlib.Path(__file__).resolve().parents[1]
_SERVICES = ROOT / "src" / "grimoire" / "services" _SERVICES = ROOT / "src" / "grimoire" / "services"
_VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue" _VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue"
# THE SIX RETRIEVAL FLOORS live in one registry, so their side of # THE RETRIEVAL FLOORS live in one registry, so their side of
# the pairing is a SURFACE NAME rather than a module constant. That is strictly # the pairing is a SURFACE NAME rather than a module constant. That is strictly
# better for this guard: the surface name is also the telemetry source, so a row # better for this guard: the surface name is also the telemetry source, so a row
# here names the same arm the readout does, and a floor that moved cannot be # here names the same arm the readout does, and a floor that moved cannot be
@@ -49,9 +49,10 @@ _VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue"
_SURFACE_PAIRS = ( _SURFACE_PAIRS = (
("auto_inject", "kbInjectThreshold"), ("auto_inject", "kbInjectThreshold"),
("write_path", "kbWritePathThreshold"), ("write_path", "kbWritePathThreshold"),
("write_path_rule", "kbRuleHintThreshold"), # The three rule hints read one bar, so one control serves them.
("pre_tool_rule", "kbToolRuleThreshold"), ("write_path_rule", "kbRuleThreshold"),
("prompt_rule", "kbPromptRuleThreshold"), ("pre_tool_rule", "kbRuleThreshold"),
("prompt_rule", "kbRuleThreshold"),
("reply_rule", "kbReplyRuleThreshold"), ("reply_rule", "kbReplyRuleThreshold"),
) )
+16 -34
View File
@@ -485,13 +485,13 @@ async def test_writepath_threshold_is_operator_tunable_and_clamped():
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_the_rule_arm_has_its_own_tunable_bar(): async def test_the_rule_arms_have_their_own_tunable_bar():
"""Settable and clamped again, for the THIRD corpus. """Settable and clamped again, for the THIRD corpus.
Separate from the code threshold above and separately settable, because the Separate from the code threshold above and separately settable, because the
two are measured against different things: 0.68 was derived from code two are measured against different things: the code bar against note
against note PROSE, and rules are short imperative technical PROSE, the rule bar against short imperative technical English — a more
English — a more homogeneous corpus whose noise floor sits higher. homogeneous corpus whose noise floor sits higher.
""" """
from grimoire.services import plugin_context as pc from grimoire.services import plugin_context as pc
@@ -524,33 +524,28 @@ async def test_the_two_write_path_bars_are_independent():
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_the_two_act_arms_read_independent_rule_bars(): async def test_both_act_arms_read_the_one_rule_bar():
"""The command arm's bar moves without the write path's. """One stored value moves the command arm and the write arm together.
The two act arms behave like different subsystems at the same number, They search the same rules, and what limits each is its budget and band.
because a code payload is long and rich where a shell command is short. A config assembler that A config assembler that read a second key into either field would split
reads one key into both fields would silently undo that, and the symptom the bar again with nothing in the settings to show it.
would be invisible: both arms would simply agree again.
""" """
from grimoire.services import plugin_context as pc from grimoire.services import plugin_context as pc
stored = {pc.RULEHINT_THRESHOLD_KEY: "0.75", pc.TOOLRULE_THRESHOLD_KEY: "0.61"} assert pc.TOOLRULE_THRESHOLD_KEY == pc.RULEHINT_THRESHOLD_KEY
stored = {pc.RULEHINT_THRESHOLD_KEY: "0.61"}
with _stored_settings(stored): with _stored_settings(stored):
cfg = await pc.get_writepath_config(1) cfg = await pc.get_writepath_config(1)
assert cfg["rule_threshold"] == 0.75 assert cfg["rule_threshold"] == 0.61
assert cfg["tool_rule_threshold"] == 0.61 assert cfg["tool_rule_threshold"] == 0.61
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_a_garbage_command_bar_falls_back_to_its_own_default(): async def test_a_garbage_rule_bar_falls_back_to_its_default():
"""Not to 0.0, and not to the write path's default. """Not to 0.0, which would attach a rule to every Bash call and every
write in the session."""
Falling back to 0.0 would attach a rule to every Bash call in the session;
falling back to the sibling's default would quietly re-merge the two bars
that are kept separate, which is the harder failure to see because the arm
keeps working.
"""
from grimoire.services import plugin_context as pc from grimoire.services import plugin_context as pc
stored = {pc.TOOLRULE_THRESHOLD_KEY: "banana"} stored = {pc.TOOLRULE_THRESHOLD_KEY: "banana"}
@@ -558,20 +553,7 @@ async def test_a_garbage_command_bar_falls_back_to_its_own_default():
cfg = await pc.get_writepath_config(1) cfg = await pc.get_writepath_config(1)
assert cfg["tool_rule_threshold"] == pc.TOOLRULE_DEFAULT_THRESHOLD assert cfg["tool_rule_threshold"] == pc.TOOLRULE_DEFAULT_THRESHOLD
assert cfg["rule_threshold"] == pc.RULEHINT_DEFAULT_THRESHOLD
def test_the_command_bar_defaults_below_the_write_path_bar():
"""A DIRECTION check, like its sibling above, and for the same reason: the
value is measured against one corpus, the relationship is not.
A shell command carries less text than a code payload and therefore scores
lower for the same relevance; at the write path's bar, consequential
commands retrieve nothing. Tuning either value stays free; inverting the
relationship would mute the command arm.
"""
from grimoire.services import plugin_context as pc
assert pc.TOOLRULE_DEFAULT_THRESHOLD < pc.RULEHINT_DEFAULT_THRESHOLD
def test_the_rule_bar_defaults_above_the_code_bar(): def test_the_rule_bar_defaults_above_the_code_bar():