align(4909d): every rule arm that annotates reads one bar, kb_rule_threshold at 0.70
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped
prompt_rule, pre_tool_rule and write_path_rule now share one settings key, RULE_FLOOR_KEY = "kb_rule_threshold", default 0.70. The reply backstop keeps its own 0.80, because there the floor is a stop bar. Moving the shared floor through tune_retrieval or the Settings form records a tuning event on every surface that reads it (set_dial, via retrieval_surfaces.sharing_floor), and the result names the surfaces it carried as `moved_with`. Settings shows one "Standing-rule confidence threshold" control where it showed three. Operator ruling (2026-10-05): one shared bar for the rule arms, "especially since we're only taking the first x number of them" -- the budget limits what is shown; the bar only decides what is worth ranking. Kept a dial. Why 0.70, from records read rather than percentiles: - write_path_rule near-misses at 0.7199-0.72 were all false positives (rule 149 against a Lidarr comment, 157 against a retention Svelte file, 121 against an Archives restore, 164 against a config test, 131 against a Go package), so the old 0.72 was not hiding real hits -- but the band does noise control on this arm, so dropping 0.02 costs little. - prompt_rule: the operator set 0.70 by hand on 2026-09-18, after a 0.60 experiment admitted junk and was reverted. 0.70 is the number with a human decision behind it. - pre_tool_rule lines near 0.68 this session were mostly noise; rising to 0.70 trims them, and the band still narrows what clears. Retired keys: kb_promptrule_threshold, kb_toolrule_threshold and kb_rulehint_threshold are no longer read. Rows an install stored under them stay in the settings table, inert. On this instance the effective bars move pre_tool_rule 0.68 -> 0.70, write_path_rule 0.72 -> 0.70, prompt_rule stays 0.70; record_release_defaults writes those per-surface release rows on boot, so the telemetry shift has an event to explain it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -159,16 +159,10 @@ const kbWritePathEnabled = ref(true);
|
||||
// code embeddings sit on a much higher similarity floor than prose, so 0.55
|
||||
// lets unrelated code through. Shares top-k, not the threshold.
|
||||
const kbWritePathThreshold = ref("0.68");
|
||||
const kbRuleHintThreshold = ref("0.72");
|
||||
// One bar for every rule hint — prompt, command and edit alike. They search
|
||||
// the same rules, and what limits each is its budget, not the bar.
|
||||
const kbRuleThreshold = ref("0.70");
|
||||
const kbCheckpointThreshold = ref("0.80");
|
||||
// The two ACT arms do not share a bar. A write-path query is a code payload; a
|
||||
// pre-tool query is a shell command, often under a dozen words — less text,
|
||||
// less signal, lower scores for the same relevance — so one shared bar makes
|
||||
// them behave like different subsystems.
|
||||
const kbToolRuleThreshold = ref("0.68");
|
||||
// And the prompt boundary is a third query shape again — the operator's own
|
||||
// prose rather than anything a tool produced.
|
||||
const kbPromptRuleThreshold = ref("0.72");
|
||||
// The reply arm: its bar is the bar at which a finished reply
|
||||
// is HELD for one read, so it sits with the checkpoint, not with the hints.
|
||||
const kbReplyRuleThreshold = ref("0.8");
|
||||
@@ -367,14 +361,10 @@ async function saveKbInject() {
|
||||
// Same `|| default` reasoning: falling back to 0 would surface every
|
||||
// snippet in the corpus on every edit, which is the failure this knob fixes.
|
||||
const wpT = asBar(kbWritePathThreshold.value, 0.68);
|
||||
// Same `|| default` reasoning again, and it bites harder here: a rule hint
|
||||
// fires on every write, so a fallback of 0 would attach a standing rule to
|
||||
// every edit in the session.
|
||||
const rhT = asBar(kbRuleHintThreshold.value, 0.72);
|
||||
// Same `|| default` guard, and the same reason: this arm fires before every
|
||||
// Bash call, so a fallback of 0 would put a rule in front of every command.
|
||||
const trT = asBar(kbToolRuleThreshold.value, 0.68);
|
||||
const prT = asBar(kbPromptRuleThreshold.value, 0.72);
|
||||
// Same `|| default` reasoning again, and it bites harder here: rule hints
|
||||
// fire on every edit and before every command, so a fallback of 0 would put
|
||||
// a standing rule beside each of them.
|
||||
const ruleT = asBar(kbRuleThreshold.value, 0.7);
|
||||
// The checkpoint bar, and the `|| default` guard matters most here of all:
|
||||
// this is the only number that can STOP a call, so a fallback of 0 would
|
||||
// hold the first command of every session behind whatever ranked first.
|
||||
@@ -409,10 +399,8 @@ async function saveKbInject() {
|
||||
kbGateNoteOverlapFloor.value = String(gateOverlap);
|
||||
kbPlanMatchThreshold.value = String(planT);
|
||||
kbWritePathThreshold.value = String(wpT);
|
||||
kbRuleHintThreshold.value = String(rhT);
|
||||
kbRuleThreshold.value = String(ruleT);
|
||||
kbCheckpointThreshold.value = String(cpT);
|
||||
kbToolRuleThreshold.value = String(trT);
|
||||
kbPromptRuleThreshold.value = String(prT);
|
||||
kbReplyRuleThreshold.value = String(ryT);
|
||||
savingKbInject.value = true;
|
||||
kbInjectSaved.value = false;
|
||||
@@ -426,20 +414,15 @@ async function saveKbInject() {
|
||||
// measurements that split them.
|
||||
kb_writepath_enabled: kbWritePathEnabled.value ? 'true' : 'false',
|
||||
kb_writepath_threshold: String(wpT),
|
||||
// A THIRD corpus with a third bar — see RULEHINT_DEFAULT_THRESHOLD
|
||||
// in services/plugin_context.py for why rules cannot share the
|
||||
// code threshold any more than code could share the prose one.
|
||||
kb_rulehint_threshold: String(rhT),
|
||||
// A third corpus with a third bar — see RULE_FLOOR_DEFAULT in
|
||||
// services/retrieval_pipeline.py for why every rule hint shares it.
|
||||
kb_rule_threshold: String(ruleT),
|
||||
// The one bar that stops a call rather than annotating it. Its own key,
|
||||
// never derived from the two rule bars above: it answers a different
|
||||
// never derived from the rule bar above: it answers a different
|
||||
// question of the same scores — not "is this worth showing" but "is the
|
||||
// corpus confident enough to be read first" — and a number that moves
|
||||
// when another moves is a number nobody can reason about.
|
||||
kb_checkpoint_threshold: String(cpT),
|
||||
// A FOURTH and FIFTH bar, and they are separate keys on purpose: one
|
||||
// number cannot serve arms whose queries are different shapes. Moving one must not move the others.
|
||||
kb_toolrule_threshold: String(trT),
|
||||
kb_promptrule_threshold: String(prT),
|
||||
// The reply backstop's own bar: it HOLDS a reply, so like the
|
||||
// checkpoint it is never derived from a hint bar.
|
||||
kb_replyrule_threshold: String(ryT),
|
||||
@@ -864,14 +847,8 @@ onMounted(async () => {
|
||||
if (allSettings.kb_checkpoint_threshold !== undefined) {
|
||||
kbCheckpointThreshold.value = allSettings.kb_checkpoint_threshold;
|
||||
}
|
||||
if (allSettings.kb_rulehint_threshold !== undefined) {
|
||||
kbRuleHintThreshold.value = allSettings.kb_rulehint_threshold;
|
||||
}
|
||||
if (allSettings.kb_toolrule_threshold !== undefined) {
|
||||
kbToolRuleThreshold.value = allSettings.kb_toolrule_threshold;
|
||||
}
|
||||
if (allSettings.kb_promptrule_threshold !== undefined) {
|
||||
kbPromptRuleThreshold.value = allSettings.kb_promptrule_threshold;
|
||||
if (allSettings.kb_rule_threshold !== undefined) {
|
||||
kbRuleThreshold.value = allSettings.kb_rule_threshold;
|
||||
}
|
||||
if (allSettings.kb_replyrule_threshold !== undefined) {
|
||||
kbReplyRuleThreshold.value = allSettings.kb_replyrule_threshold;
|
||||
@@ -1772,10 +1749,10 @@ async function deleteUser(userId: number) {
|
||||
<p class="field-hint">How many snippets and issues one edit may be offered (1–10). Reach for this rather than the threshold when the hint feels long: lowering it keeps the best matches and drops the tail, while raising the bar above throws away good matches along with weak ones.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-rulehint-threshold">Standing-rule confidence threshold (0–1)</label>
|
||||
<label for="kb-rule-threshold">Standing-rule confidence threshold (0–1)</label>
|
||||
<input
|
||||
id="kb-rulehint-threshold"
|
||||
v-model="kbRuleHintThreshold"
|
||||
id="kb-rule-threshold"
|
||||
v-model="kbRuleThreshold"
|
||||
type="number"
|
||||
min="0"
|
||||
max="1"
|
||||
@@ -1784,14 +1761,14 @@ async function deleteUser(userId: number) {
|
||||
style="max-width: 8rem"
|
||||
/>
|
||||
<p class="field-hint">
|
||||
The same hint can mention a standing rule whose trigger resembles the
|
||||
code being written. <em>Every</em> rule is eligible — nothing is
|
||||
preloaded any more, so this is the only way a rule reaches a write.
|
||||
Stricter than the threshold above, because there are far fewer rules
|
||||
than snippets: with a small set something always ranks first, so the
|
||||
bar has to carry more of the judgement. If rules arrive unread, or one
|
||||
you needed never showed up, that is Claude's to notice and correct
|
||||
from the telemetry — and the budget below is usually the better lever.
|
||||
The bar for a standing rule or preference to be offered at all,
|
||||
whether against the code being written, a command about to run, or
|
||||
what you just said. <em>Every</em> rule is eligible — nothing is
|
||||
preloaded, so this is how a rule reaches the work. One bar serves all
|
||||
three: they search the same rules, and how many each may show is set
|
||||
by its own budget below, which is usually the better lever. If rules
|
||||
arrive unread, or one you needed never showed up, that is Claude's to
|
||||
notice and correct from the telemetry.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
@@ -1834,26 +1811,6 @@ async function deleteUser(userId: number) {
|
||||
only — file edits are never held.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-toolrule-threshold">Command confidence threshold (0–1)</label>
|
||||
<input
|
||||
id="kb-toolrule-threshold"
|
||||
v-model="kbToolRuleThreshold"
|
||||
type="number"
|
||||
min="0"
|
||||
max="1"
|
||||
step="0.01"
|
||||
class="fs-input input"
|
||||
style="max-width: 8rem"
|
||||
/>
|
||||
<p class="field-hint">
|
||||
The bar for a rule surfacing before a <em>command</em> runs, where the
|
||||
query is the command text rather than code. Lower than the one above
|
||||
on purpose: a shell command is short, so it scores lower for the same
|
||||
relevance — at a shared bar this arm spoke on 2% of calls against the
|
||||
write path's 37%.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-toolrule-topk">Max rules per command</label>
|
||||
<input
|
||||
@@ -1868,26 +1825,6 @@ async function deleteUser(userId: number) {
|
||||
/>
|
||||
<p class="field-hint">How many standing rules one command may be shown (1–10). This is the busiest arm there is — it fires before every command — so its budget is the one most worth keeping small.</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-promptrule-threshold">Prompt confidence threshold (0–1)</label>
|
||||
<input
|
||||
id="kb-promptrule-threshold"
|
||||
v-model="kbPromptRuleThreshold"
|
||||
type="number"
|
||||
min="0"
|
||||
max="1"
|
||||
step="0.01"
|
||||
class="fs-input input"
|
||||
style="max-width: 8rem"
|
||||
/>
|
||||
<p class="field-hint">
|
||||
The bar for a rule or preference surfacing against <em>what you just
|
||||
said</em>, before anything is done. It is the only moment that reaches
|
||||
a rule about how to answer rather than how to act, so it runs once a
|
||||
turn rather than once a tool call. A third query shape — your prose,
|
||||
not a command or a file — which is why it carries its own number.
|
||||
</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="kb-promptrule-topk">Max rules per prompt</label>
|
||||
<input
|
||||
|
||||
@@ -144,7 +144,7 @@ WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
|
||||
WRITEPATH_DEFAULT_ENABLED = True
|
||||
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
|
||||
|
||||
# The standing-rule arm gets its own bar, for a third corpus: 0.68 above is
|
||||
# The standing-rule arms get their own bar, for a third corpus: 0.68 above is
|
||||
# calibrated on code against note PROSE, not code against RULE TEXT.
|
||||
#
|
||||
# THE STRUCTURAL ARGUMENT, which holds on any install, including one with six
|
||||
@@ -155,64 +155,27 @@ WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
|
||||
# pool always returns something, so "the best match cleared the bar"
|
||||
# drifts from "a good match exists" toward "N things were ranked". A bar
|
||||
# calibrated for best-of-thousands is cleared by best-of-forty as
|
||||
# arithmetic rather than relevance. (The arms search every tier, which
|
||||
# makes the pool larger; the bar still errs toward silence on purpose.)
|
||||
# arithmetic rather than relevance.
|
||||
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS
|
||||
# corpus than note prose, and a more homogeneous corpus has a HIGHER
|
||||
# noise floor than prose's 0.55-0.63.
|
||||
#
|
||||
# WHY 0.72 AND NOT A NUMBER OFF A HISTOGRAM. The exact offset between prose's
|
||||
# floor and rule-text's is not derivable in general — it depends on how an
|
||||
# install writes its rules — so the default errs deliberately toward SILENCE
|
||||
# rather than toward recall, on an asymmetry that is itself structural: this
|
||||
# hint fires on EVERY write. A missed rule is recoverable, because the rule is
|
||||
# still in Grimoire and the agent can search it. A hint that cries wolf is not:
|
||||
# it teaches the reader to skip the whole block, and the surface is lost along
|
||||
# with the true positives it would have carried.
|
||||
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The write path, the command arm
|
||||
# and the prompt arm read the same key (`RULE_FLOOR_KEY` in retrieval_pipeline):
|
||||
# they search the same rules, and what limits each one is its budget and band.
|
||||
#
|
||||
# TUNE IT FROM YOUR OWN INSTANCE: `retrieval_telemetry` reports
|
||||
# `rule_usage.pull_through`. Raise this if rules arrive unread; lower it if
|
||||
# rules you needed never arrived. What would RETIRE it: a cross-encoder rerank,
|
||||
# which would make a similarity bar the wrong control entirely.
|
||||
#
|
||||
# This bar is the WRITE-PATH arm's. A code payload is long and rich, which is
|
||||
# the case 0.72 suits; the command arm has its own bar below.
|
||||
# `rule_usage.pull_through` and `near_miss_samples`. Raise it if rules arrive
|
||||
# unread; lower it if rules you needed never arrived. What would RETIRE it: a
|
||||
# cross-encoder rerank, which would make a similarity bar the wrong control
|
||||
# entirely.
|
||||
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
|
||||
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
|
||||
|
||||
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S.
|
||||
#
|
||||
# A write-path query is a code payload, long and rich, while a pre-tool query
|
||||
# is a shell command — often under a dozen words. Less text, less signal, lower
|
||||
# scores for the same relevance. At the write path's 0.72 the command arm
|
||||
# turns nearly everything away, with its refused mass piled just under the
|
||||
# line, which is the shape a bar set too high leaves.
|
||||
#
|
||||
# WHAT 0.68 IS CALIBRATED AGAINST. Replayed queries, consequential acts
|
||||
# against innocuous ones, scored against a rule corpus with symptom-bearing
|
||||
# triggers:
|
||||
#
|
||||
# 0.7571 git push origin dev consequential
|
||||
# 0.7245 cd ...; git fetch; git add -A consequential
|
||||
# 0.7193 git pull --rebase origin dev consequential
|
||||
# 0.6850 docker compose up -d consequential
|
||||
# ---------------------------------------- 0.68
|
||||
# 0.6735 wc -l src/*.py && date innocuous
|
||||
# 0.6544 grep -rn useState src/ innocuous
|
||||
# 0.6099 sed -n '120,160p' package.json innocuous
|
||||
# 0.6056 ls -la && cat README.md innocuous
|
||||
#
|
||||
# At 0.72 three of the four consequential acts would retrieve nothing, though
|
||||
# the right rules ranked first for each.
|
||||
#
|
||||
# THE SEPARATION IS NARROW (about 0.01), and that is a caveat, not a result.
|
||||
# Eight probes set a direction; they do not settle a number.
|
||||
# `near_miss_samples` on real traffic is what settles it, and this is the bar
|
||||
# to re-read first.
|
||||
#
|
||||
# WHY A LOWER BAR IS SAFE HERE. The band does noise control downstream: a marginal hit that
|
||||
# clears the bar still has to score within `_RULEHINT_BAND` of the top to be
|
||||
# rendered. The bar's job shrank, so the bar can.
|
||||
# THE COMMAND ARM READS THE SAME BAR. A shell command is short, so it scores
|
||||
# lower than a code payload for the same relevance and the shared bar admits
|
||||
# less here. The band does noise control downstream: a hit that clears the bar
|
||||
# still has to score within `_RULEHINT_BAND` of the top to be rendered.
|
||||
#
|
||||
# The innocuous scores near the line are largely CROSS-PROJECT BLEED rather
|
||||
# than bad ranking: another project's shell-command rule matching a shell
|
||||
@@ -478,17 +441,11 @@ def _autoinject_query(prompt: str, context: str) -> str:
|
||||
# The operator's message is the only query that exists before one is composed,
|
||||
# and this arm is what runs against it.
|
||||
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
|
||||
# INHERITED FROM THE ACT ARMS. 0.72 is tuned against
|
||||
# code and shell commands. An operator's prose is a different query shape
|
||||
# against the same documents, and nothing yet says the two distributions line
|
||||
# up — triggers are written in the vocabulary of the MOMENT, which for most
|
||||
# rules is act vocabulary, so prose may well score lower across the board.
|
||||
#
|
||||
# Starting at the act arms' number anyway is deliberate: it is the only value
|
||||
# with evidence behind it, and guessing lower would put an unmeasured bar in
|
||||
# front of a corpus that binds. Every call is logged under `prompt_rule`, so
|
||||
# real traffic settles it — read `near_miss_samples` before moving this, not
|
||||
# the percentile alone.
|
||||
# THE SAME BAR AS THE ACT ARMS. An operator's prose is a different query shape
|
||||
# against the same documents, and triggers are written in the vocabulary of
|
||||
# the moment, which for most rules is act vocabulary. Every call is logged
|
||||
# under `prompt_rule`, so read `near_miss_samples` before moving the shared
|
||||
# bar, not the percentile alone.
|
||||
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
|
||||
|
||||
# ANCHORED ON THIS HOOK'S BUDGET rather than the act arms'. This arm fires
|
||||
@@ -1171,9 +1128,8 @@ async def get_writepath_config(user_id: int) -> dict:
|
||||
independent pairs, resolved from the registry.
|
||||
|
||||
Each floor's reason is recorded where its default is: WRITEPATH (code
|
||||
embeddings sit on a much higher similarity floor than prose), RULEHINT (a
|
||||
third corpus again), TOOLRULE (a shell command is a different query shape
|
||||
from a code payload and scores lower for the same relevance).
|
||||
embeddings sit on a much higher similarity floor than prose) and RULEHINT
|
||||
(a third corpus again, whose one bar both rule arms read).
|
||||
|
||||
THE BUDGET IS SPLIT TOO. "How many titles at once" does not mean the same
|
||||
thing on both surfaces: this arm fires before every Write and Edit while
|
||||
|
||||
@@ -460,6 +460,20 @@ class RuleArm:
|
||||
declared: Declared | None = None
|
||||
|
||||
|
||||
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The prompt, command and write
|
||||
# arms search the same corpus, and what limits what they show is the budget
|
||||
# and the band, not the floor. The floor only answers "is this worth ranking
|
||||
# at all", and that question has one answer per corpus, not one per query
|
||||
# shape — so the three share one settings key, and moving it moves all three.
|
||||
# `set_dial` records the move on every surface that shares the key.
|
||||
#
|
||||
# Set low on purpose: a hit under it is never shown, so a bar set too high
|
||||
# costs a rule that binds, while one set a little low costs a line the band
|
||||
# usually drops. The reply backstop keeps its own number, because there the
|
||||
# floor is a stop bar and not a hint bar.
|
||||
RULE_FLOOR_KEY = "kb_rule_threshold"
|
||||
RULE_FLOOR_DEFAULT = 0.70
|
||||
|
||||
# The arms differ on purpose where these flags differ. Whether the prompt arm
|
||||
# should band, and whether the act arms should reserve a preference, are open
|
||||
# questions to settle on telemetry, not by symmetry.
|
||||
@@ -468,8 +482,8 @@ PROMPT_RULE = RuleArm(
|
||||
preference_slot=True,
|
||||
tuning=Surface(
|
||||
name="prompt_rule",
|
||||
floor_key="kb_promptrule_threshold",
|
||||
floor_default=0.72,
|
||||
floor_key=RULE_FLOOR_KEY,
|
||||
floor_default=RULE_FLOOR_DEFAULT,
|
||||
budget_key="kb_promptrule_top_k",
|
||||
budget_default=3,
|
||||
asks="the operator's message, against rule triggers",
|
||||
@@ -483,8 +497,8 @@ PRE_TOOL_RULE = RuleArm(
|
||||
preference_slot=False,
|
||||
tuning=Surface(
|
||||
name="pre_tool_rule",
|
||||
floor_key="kb_toolrule_threshold",
|
||||
floor_default=0.68,
|
||||
floor_key=RULE_FLOOR_KEY,
|
||||
floor_default=RULE_FLOOR_DEFAULT,
|
||||
budget_key="kb_toolrule_top_k",
|
||||
budget_default=5,
|
||||
asks="the command about to run, against rule triggers",
|
||||
@@ -498,8 +512,8 @@ WRITE_PATH_RULE = RuleArm(
|
||||
preference_slot=False,
|
||||
tuning=Surface(
|
||||
name="write_path_rule",
|
||||
floor_key="kb_rulehint_threshold",
|
||||
floor_default=0.72,
|
||||
floor_key=RULE_FLOOR_KEY,
|
||||
floor_default=RULE_FLOOR_DEFAULT,
|
||||
budget_key="kb_rulehint_top_k",
|
||||
budget_default=5,
|
||||
asks="the code being written, against rule triggers",
|
||||
|
||||
@@ -110,6 +110,9 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
|
||||
|
||||
Derived rather than listed so a seventh surface is covered the moment it is
|
||||
added here, which is the only way this stays true.
|
||||
|
||||
A floor key several surfaces share resolves to the first of them;
|
||||
`set_dial` records the move on every surface in `sharing_floor`.
|
||||
"""
|
||||
for surface in SURFACES.values():
|
||||
if key == surface.floor_key:
|
||||
@@ -119,6 +122,16 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
|
||||
return None
|
||||
|
||||
|
||||
def sharing_floor(name: str) -> list[str]:
|
||||
"""Every surface that reads the same floor key as `name`, itself included.
|
||||
|
||||
The rule arms share one bar, so moving one surface's floor moves its
|
||||
siblings, and each of them needs the event that says so.
|
||||
"""
|
||||
key = get_surface(name).floor_key
|
||||
return [n for n, s in SURFACES.items() if s.floor_key == key]
|
||||
|
||||
|
||||
async def floor_for(user_id: int, name: str) -> float:
|
||||
"""This install's current floor for a surface, clamped to [0, 1]."""
|
||||
s = get_surface(name)
|
||||
|
||||
@@ -44,6 +44,7 @@ from grimoire.services.retrieval_surfaces import (
|
||||
budget_for,
|
||||
floor_for,
|
||||
get_surface,
|
||||
sharing_floor,
|
||||
surface_names,
|
||||
)
|
||||
from grimoire.services.settings import set_setting
|
||||
@@ -257,6 +258,10 @@ async def set_dial(
|
||||
"reason": text, "actor": actor, "unchanged": True,
|
||||
}
|
||||
|
||||
# A shared floor moves every surface that reads it, and each one's history
|
||||
# has to show the move, or its telemetry changes with no event to explain it.
|
||||
moved = sharing_floor(surface) if dial == "floor" else [surface]
|
||||
|
||||
await set_setting(user_id, key, stored)
|
||||
async with async_session() as session:
|
||||
# Stamped with the space this number was chosen in. Read at
|
||||
@@ -264,12 +269,13 @@ async def set_dial(
|
||||
# the caller recording it are the same call, so there is no window in
|
||||
# which they could disagree.
|
||||
stamp = calibration_stamp()
|
||||
session.add(RetrievalTuningEvent(
|
||||
user_id=user_id, surface=surface, dial=dial,
|
||||
old_value=old, new_value=applied, actor=actor, reason=text,
|
||||
embedding_model=stamp["embedding_model"],
|
||||
shape_version=stamp["shape_version"],
|
||||
))
|
||||
for name in moved:
|
||||
session.add(RetrievalTuningEvent(
|
||||
user_id=user_id, surface=name, dial=dial,
|
||||
old_value=old, new_value=applied, actor=actor, reason=text,
|
||||
embedding_model=stamp["embedding_model"],
|
||||
shape_version=stamp["shape_version"],
|
||||
))
|
||||
await session.commit()
|
||||
|
||||
return {
|
||||
@@ -286,6 +292,9 @@ async def set_dial(
|
||||
# Always present, both ways round: a caller that has to test for the
|
||||
# key's absence to learn the answer will eventually forget to.
|
||||
"unchanged": False,
|
||||
# The other surfaces this move carried with it: a shared floor is one
|
||||
# number, and a caller tuning one rule arm has tuned them all.
|
||||
"moved_with": [n for n in moved if n != surface],
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -71,13 +71,29 @@ def test_every_surface_name_is_a_real_telemetry_source():
|
||||
)
|
||||
|
||||
|
||||
def test_no_two_surfaces_share_a_settings_key():
|
||||
"""One dial, one label."""
|
||||
floors = [s.floor_key for s in rs.SURFACES.values()]
|
||||
def test_a_shared_settings_key_is_one_bar_with_one_default():
|
||||
"""One dial, one label. Surfaces may share a floor key, as the rule arms
|
||||
do, but then they must ship the same default: otherwise the one number
|
||||
means something different on each surface until somebody sets it."""
|
||||
by_key: dict[str, set[float]] = {}
|
||||
for s in rs.SURFACES.values():
|
||||
by_key.setdefault(s.floor_key, set()).add(s.floor_default)
|
||||
split = {k: v for k, v in by_key.items() if len(v) > 1}
|
||||
assert not split, f"a shared floor key ships two defaults: {split}"
|
||||
|
||||
budgets = [s.budget_key for s in rs.SURFACES.values()]
|
||||
assert len(set(floors)) == len(floors), f"duplicate floor key in {floors}"
|
||||
assert len(set(budgets)) == len(budgets), f"duplicate budget key in {budgets}"
|
||||
assert not (set(floors) & set(budgets)), "a floor key doubles as a budget key"
|
||||
assert not (set(by_key) & set(budgets)), "a floor key doubles as a budget key"
|
||||
|
||||
|
||||
def test_the_rule_arms_that_annotate_share_one_bar_and_the_reply_stop_does_not():
|
||||
from grimoire.services.retrieval_pipeline import RULE_FLOOR_KEY
|
||||
|
||||
assert sorted(rs.sharing_floor("prompt_rule")) == [
|
||||
"pre_tool_rule", "prompt_rule", "write_path_rule",
|
||||
]
|
||||
assert rs.SURFACES["prompt_rule"].floor_key == RULE_FLOOR_KEY
|
||||
assert rs.sharing_floor("reply_rule") == ["reply_rule"]
|
||||
|
||||
|
||||
def test_every_surface_says_what_it_asks_over_what_and_how_often():
|
||||
|
||||
@@ -86,9 +86,9 @@ async def test_a_good_change_writes_the_setting_and_the_event():
|
||||
|
||||
setter.assert_awaited_once()
|
||||
_uid, key, stored = setter.await_args.args
|
||||
assert key == "kb_promptrule_threshold" and stored == "0.66"
|
||||
assert key == "kb_rule_threshold" and stored == "0.66"
|
||||
|
||||
event = session.add.call_args.args[0]
|
||||
event = session.add.call_args_list[0].args[0]
|
||||
assert event.surface == "prompt_rule" and event.dial == "floor"
|
||||
# The before-value is what makes the history answerable.
|
||||
assert event.old_value == 0.72 and event.new_value == 0.66
|
||||
@@ -97,6 +97,37 @@ async def test_a_good_change_writes_the_setting_and_the_event():
|
||||
assert out["clamped"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_shared_floor_records_the_move_on_every_surface_that_reads_it():
|
||||
"""One key, one write, an event per surface. A rule arm whose bar moved
|
||||
with no event of its own would show a telemetry shift nothing explains."""
|
||||
session, ctx = _patches(floor=0.70)
|
||||
with ctx[0], patch.object(rt, "set_setting", AsyncMock()) as setter, \
|
||||
ctx[2], ctx[3]:
|
||||
out = await rt.set_dial(
|
||||
1, "pre_tool_rule", "floor", 0.74,
|
||||
reason="read the 10 highest declines: every one was a real miss",
|
||||
)
|
||||
|
||||
setter.assert_awaited_once()
|
||||
recorded = {c.args[0].surface for c in session.add.call_args_list}
|
||||
assert recorded == {"pre_tool_rule", "prompt_rule", "write_path_rule"}
|
||||
assert sorted(out["moved_with"]) == ["prompt_rule", "write_path_rule"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_budget_moves_only_its_own_surface():
|
||||
session, ctx = _patches(budget=3)
|
||||
with ctx[0], patch.object(rt, "set_setting", AsyncMock()), ctx[2], ctx[3]:
|
||||
out = await rt.set_dial(
|
||||
1, "prompt_rule", "budget", 4,
|
||||
reason="on-point hits sit at ranks 4 and 5 as often as rank 1",
|
||||
)
|
||||
|
||||
assert [c.args[0].surface for c in session.add.call_args_list] == ["prompt_rule"]
|
||||
assert out["moved_with"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("dial, sent, applied", [
|
||||
("floor", 1.4, 1.0),
|
||||
|
||||
@@ -76,10 +76,15 @@ def test_the_route_does_not_take_the_actor_from_the_caller():
|
||||
def test_every_registry_key_is_recognised_as_a_dial():
|
||||
"""Derived from the registry, both directions. A seventh surface added
|
||||
without a row here is a floor that can be written silently again."""
|
||||
from grimoire.services.retrieval_surfaces import SURFACES, dial_for_key
|
||||
from grimoire.services.retrieval_surfaces import (
|
||||
SURFACES, dial_for_key, sharing_floor,
|
||||
)
|
||||
|
||||
for name, surface in SURFACES.items():
|
||||
assert dial_for_key(surface.floor_key) == (name, "floor")
|
||||
# A shared floor resolves to one of the surfaces that read it, and
|
||||
# `set_dial` records the move on all of them.
|
||||
owner, dial = dial_for_key(surface.floor_key)
|
||||
assert dial == "floor" and name in sharing_floor(owner)
|
||||
assert dial_for_key(surface.budget_key) == (name, "budget")
|
||||
|
||||
|
||||
|
||||
@@ -41,7 +41,7 @@ ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
_SERVICES = ROOT / "src" / "grimoire" / "services"
|
||||
_VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue"
|
||||
|
||||
# THE SIX RETRIEVAL FLOORS live in one registry, so their side of
|
||||
# THE RETRIEVAL FLOORS live in one registry, so their side of
|
||||
# the pairing is a SURFACE NAME rather than a module constant. That is strictly
|
||||
# better for this guard: the surface name is also the telemetry source, so a row
|
||||
# here names the same arm the readout does, and a floor that moved cannot be
|
||||
@@ -49,9 +49,10 @@ _VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue"
|
||||
_SURFACE_PAIRS = (
|
||||
("auto_inject", "kbInjectThreshold"),
|
||||
("write_path", "kbWritePathThreshold"),
|
||||
("write_path_rule", "kbRuleHintThreshold"),
|
||||
("pre_tool_rule", "kbToolRuleThreshold"),
|
||||
("prompt_rule", "kbPromptRuleThreshold"),
|
||||
# The three rule hints read one bar, so one control serves them.
|
||||
("write_path_rule", "kbRuleThreshold"),
|
||||
("pre_tool_rule", "kbRuleThreshold"),
|
||||
("prompt_rule", "kbRuleThreshold"),
|
||||
("reply_rule", "kbReplyRuleThreshold"),
|
||||
)
|
||||
|
||||
|
||||
@@ -485,13 +485,13 @@ async def test_writepath_threshold_is_operator_tunable_and_clamped():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_rule_arm_has_its_own_tunable_bar():
|
||||
async def test_the_rule_arms_have_their_own_tunable_bar():
|
||||
"""Settable and clamped again, for the THIRD corpus.
|
||||
|
||||
Separate from the code threshold above and separately settable, because the
|
||||
two are measured against different things: 0.68 was derived from code
|
||||
against note PROSE, and rules are short imperative technical
|
||||
English — a more homogeneous corpus whose noise floor sits higher.
|
||||
two are measured against different things: the code bar against note
|
||||
PROSE, the rule bar against short imperative technical English — a more
|
||||
homogeneous corpus whose noise floor sits higher.
|
||||
"""
|
||||
from grimoire.services import plugin_context as pc
|
||||
|
||||
@@ -524,33 +524,28 @@ async def test_the_two_write_path_bars_are_independent():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_two_act_arms_read_independent_rule_bars():
|
||||
"""The command arm's bar moves without the write path's.
|
||||
async def test_both_act_arms_read_the_one_rule_bar():
|
||||
"""One stored value moves the command arm and the write arm together.
|
||||
|
||||
The two act arms behave like different subsystems at the same number,
|
||||
because a code payload is long and rich where a shell command is short. A config assembler that
|
||||
reads one key into both fields would silently undo that, and the symptom
|
||||
would be invisible: both arms would simply agree again.
|
||||
They search the same rules, and what limits each is its budget and band.
|
||||
A config assembler that read a second key into either field would split
|
||||
the bar again with nothing in the settings to show it.
|
||||
"""
|
||||
from grimoire.services import plugin_context as pc
|
||||
|
||||
stored = {pc.RULEHINT_THRESHOLD_KEY: "0.75", pc.TOOLRULE_THRESHOLD_KEY: "0.61"}
|
||||
assert pc.TOOLRULE_THRESHOLD_KEY == pc.RULEHINT_THRESHOLD_KEY
|
||||
stored = {pc.RULEHINT_THRESHOLD_KEY: "0.61"}
|
||||
with _stored_settings(stored):
|
||||
cfg = await pc.get_writepath_config(1)
|
||||
|
||||
assert cfg["rule_threshold"] == 0.75
|
||||
assert cfg["rule_threshold"] == 0.61
|
||||
assert cfg["tool_rule_threshold"] == 0.61
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_garbage_command_bar_falls_back_to_its_own_default():
|
||||
"""Not to 0.0, and not to the write path's default.
|
||||
|
||||
Falling back to 0.0 would attach a rule to every Bash call in the session;
|
||||
falling back to the sibling's default would quietly re-merge the two bars
|
||||
that are kept separate, which is the harder failure to see because the arm
|
||||
keeps working.
|
||||
"""
|
||||
async def test_a_garbage_rule_bar_falls_back_to_its_default():
|
||||
"""Not to 0.0, which would attach a rule to every Bash call and every
|
||||
write in the session."""
|
||||
from grimoire.services import plugin_context as pc
|
||||
|
||||
stored = {pc.TOOLRULE_THRESHOLD_KEY: "banana"}
|
||||
@@ -558,20 +553,7 @@ async def test_a_garbage_command_bar_falls_back_to_its_own_default():
|
||||
cfg = await pc.get_writepath_config(1)
|
||||
|
||||
assert cfg["tool_rule_threshold"] == pc.TOOLRULE_DEFAULT_THRESHOLD
|
||||
|
||||
|
||||
def test_the_command_bar_defaults_below_the_write_path_bar():
|
||||
"""A DIRECTION check, like its sibling above, and for the same reason: the
|
||||
value is measured against one corpus, the relationship is not.
|
||||
|
||||
A shell command carries less text than a code payload and therefore scores
|
||||
lower for the same relevance; at the write path's bar, consequential
|
||||
commands retrieve nothing. Tuning either value stays free; inverting the
|
||||
relationship would mute the command arm.
|
||||
"""
|
||||
from grimoire.services import plugin_context as pc
|
||||
|
||||
assert pc.TOOLRULE_DEFAULT_THRESHOLD < pc.RULEHINT_DEFAULT_THRESHOLD
|
||||
assert cfg["rule_threshold"] == pc.RULEHINT_DEFAULT_THRESHOLD
|
||||
|
||||
|
||||
def test_the_rule_bar_defaults_above_the_code_bar():
|
||||
|
||||
Reference in New Issue
Block a user