align(4909d): every rule arm that annotates reads one bar, kb_rule_threshold at 0.70
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped

prompt_rule, pre_tool_rule and write_path_rule now share one settings key,
RULE_FLOOR_KEY = "kb_rule_threshold", default 0.70. The reply backstop keeps
its own 0.80, because there the floor is a stop bar. Moving the shared floor
through tune_retrieval or the Settings form records a tuning event on every
surface that reads it (set_dial, via retrieval_surfaces.sharing_floor), and
the result names the surfaces it carried as `moved_with`. Settings shows one
"Standing-rule confidence threshold" control where it showed three.

Operator ruling (2026-10-05): one shared bar for the rule arms, "especially
since we're only taking the first x number of them" -- the budget limits
what is shown; the bar only decides what is worth ranking. Kept a dial.

Why 0.70, from records read rather than percentiles:
- write_path_rule near-misses at 0.7199-0.72 were all false positives
  (rule 149 against a Lidarr comment, 157 against a retention Svelte file,
  121 against an Archives restore, 164 against a config test, 131 against a
  Go package), so the old 0.72 was not hiding real hits -- but the band does
  noise control on this arm, so dropping 0.02 costs little.
- prompt_rule: the operator set 0.70 by hand on 2026-09-18, after a 0.60
  experiment admitted junk and was reverted. 0.70 is the number with a human
  decision behind it.
- pre_tool_rule lines near 0.68 this session were mostly noise; rising to
  0.70 trims them, and the band still narrows what clears.

Retired keys: kb_promptrule_threshold, kb_toolrule_threshold and
kb_rulehint_threshold are no longer read. Rows an install stored under them
stay in the settings table, inert. On this instance the effective bars move
pre_tool_rule 0.68 -> 0.70, write_path_rule 0.72 -> 0.70, prompt_rule stays
0.70; record_release_defaults writes those per-surface release rows on boot,
so the telemetry shift has an event to explain it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-22 18:32:09 -04:00
co-authored by Claude Opus 5.5
parent f77d18cb63
commit c8ede427ca
10 changed files with 175 additions and 211 deletions
+20 -64
View File
@@ -144,7 +144,7 @@ WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
WRITEPATH_DEFAULT_ENABLED = True
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
# The standing-rule arm gets its own bar, for a third corpus: 0.68 above is
# The standing-rule arms get their own bar, for a third corpus: 0.68 above is
# calibrated on code against note PROSE, not code against RULE TEXT.
#
# THE STRUCTURAL ARGUMENT, which holds on any install, including one with six
@@ -155,64 +155,27 @@ WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
# pool always returns something, so "the best match cleared the bar"
# drifts from "a good match exists" toward "N things were ranked". A bar
# calibrated for best-of-thousands is cleared by best-of-forty as
# arithmetic rather than relevance. (The arms search every tier, which
# makes the pool larger; the bar still errs toward silence on purpose.)
# arithmetic rather than relevance.
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS
# corpus than note prose, and a more homogeneous corpus has a HIGHER
# noise floor than prose's 0.55-0.63.
#
# WHY 0.72 AND NOT A NUMBER OFF A HISTOGRAM. The exact offset between prose's
# floor and rule-text's is not derivable in general — it depends on how an
# install writes its rules — so the default errs deliberately toward SILENCE
# rather than toward recall, on an asymmetry that is itself structural: this
# hint fires on EVERY write. A missed rule is recoverable, because the rule is
# still in Grimoire and the agent can search it. A hint that cries wolf is not:
# it teaches the reader to skip the whole block, and the surface is lost along
# with the true positives it would have carried.
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The write path, the command arm
# and the prompt arm read the same key (`RULE_FLOOR_KEY` in retrieval_pipeline):
# they search the same rules, and what limits each one is its budget and band.
#
# TUNE IT FROM YOUR OWN INSTANCE: `retrieval_telemetry` reports
# `rule_usage.pull_through`. Raise this if rules arrive unread; lower it if
# rules you needed never arrived. What would RETIRE it: a cross-encoder rerank,
# which would make a similarity bar the wrong control entirely.
#
# This bar is the WRITE-PATH arm's. A code payload is long and rich, which is
# the case 0.72 suits; the command arm has its own bar below.
# `rule_usage.pull_through` and `near_miss_samples`. Raise it if rules arrive
# unread; lower it if rules you needed never arrived. What would RETIRE it: a
# cross-encoder rerank, which would make a similarity bar the wrong control
# entirely.
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S.
#
# A write-path query is a code payload, long and rich, while a pre-tool query
# is a shell command — often under a dozen words. Less text, less signal, lower
# scores for the same relevance. At the write path's 0.72 the command arm
# turns nearly everything away, with its refused mass piled just under the
# line, which is the shape a bar set too high leaves.
#
# WHAT 0.68 IS CALIBRATED AGAINST. Replayed queries, consequential acts
# against innocuous ones, scored against a rule corpus with symptom-bearing
# triggers:
#
# 0.7571 git push origin dev consequential
# 0.7245 cd ...; git fetch; git add -A consequential
# 0.7193 git pull --rebase origin dev consequential
# 0.6850 docker compose up -d consequential
# ---------------------------------------- 0.68
# 0.6735 wc -l src/*.py && date innocuous
# 0.6544 grep -rn useState src/ innocuous
# 0.6099 sed -n '120,160p' package.json innocuous
# 0.6056 ls -la && cat README.md innocuous
#
# At 0.72 three of the four consequential acts would retrieve nothing, though
# the right rules ranked first for each.
#
# THE SEPARATION IS NARROW (about 0.01), and that is a caveat, not a result.
# Eight probes set a direction; they do not settle a number.
# `near_miss_samples` on real traffic is what settles it, and this is the bar
# to re-read first.
#
# WHY A LOWER BAR IS SAFE HERE. The band does noise control downstream: a marginal hit that
# clears the bar still has to score within `_RULEHINT_BAND` of the top to be
# rendered. The bar's job shrank, so the bar can.
# THE COMMAND ARM READS THE SAME BAR. A shell command is short, so it scores
# lower than a code payload for the same relevance and the shared bar admits
# less here. The band does noise control downstream: a hit that clears the bar
# still has to score within `_RULEHINT_BAND` of the top to be rendered.
#
# The innocuous scores near the line are largely CROSS-PROJECT BLEED rather
# than bad ranking: another project's shell-command rule matching a shell
@@ -478,17 +441,11 @@ def _autoinject_query(prompt: str, context: str) -> str:
# The operator's message is the only query that exists before one is composed,
# and this arm is what runs against it.
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
# INHERITED FROM THE ACT ARMS. 0.72 is tuned against
# code and shell commands. An operator's prose is a different query shape
# against the same documents, and nothing yet says the two distributions line
# up — triggers are written in the vocabulary of the MOMENT, which for most
# rules is act vocabulary, so prose may well score lower across the board.
#
# Starting at the act arms' number anyway is deliberate: it is the only value
# with evidence behind it, and guessing lower would put an unmeasured bar in
# front of a corpus that binds. Every call is logged under `prompt_rule`, so
# real traffic settles it — read `near_miss_samples` before moving this, not
# the percentile alone.
# THE SAME BAR AS THE ACT ARMS. An operator's prose is a different query shape
# against the same documents, and triggers are written in the vocabulary of
# the moment, which for most rules is act vocabulary. Every call is logged
# under `prompt_rule`, so read `near_miss_samples` before moving the shared
# bar, not the percentile alone.
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
# ANCHORED ON THIS HOOK'S BUDGET rather than the act arms'. This arm fires
@@ -1171,9 +1128,8 @@ async def get_writepath_config(user_id: int) -> dict:
independent pairs, resolved from the registry.
Each floor's reason is recorded where its default is: WRITEPATH (code
embeddings sit on a much higher similarity floor than prose), RULEHINT (a
third corpus again), TOOLRULE (a shell command is a different query shape
from a code payload and scores lower for the same relevance).
embeddings sit on a much higher similarity floor than prose) and RULEHINT
(a third corpus again, whose one bar both rule arms read).
THE BUDGET IS SPLIT TOO. "How many titles at once" does not mean the same
thing on both surfaces: this arm fires before every Write and Edit while
+20 -6
View File
@@ -460,6 +460,20 @@ class RuleArm:
declared: Declared | None = None
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The prompt, command and write
# arms search the same corpus, and what limits what they show is the budget
# and the band, not the floor. The floor only answers "is this worth ranking
# at all", and that question has one answer per corpus, not one per query
# shape — so the three share one settings key, and moving it moves all three.
# `set_dial` records the move on every surface that shares the key.
#
# Set low on purpose: a hit under it is never shown, so a bar set too high
# costs a rule that binds, while one set a little low costs a line the band
# usually drops. The reply backstop keeps its own number, because there the
# floor is a stop bar and not a hint bar.
RULE_FLOOR_KEY = "kb_rule_threshold"
RULE_FLOOR_DEFAULT = 0.70
# The arms differ on purpose where these flags differ. Whether the prompt arm
# should band, and whether the act arms should reserve a preference, are open
# questions to settle on telemetry, not by symmetry.
@@ -468,8 +482,8 @@ PROMPT_RULE = RuleArm(
preference_slot=True,
tuning=Surface(
name="prompt_rule",
floor_key="kb_promptrule_threshold",
floor_default=0.72,
floor_key=RULE_FLOOR_KEY,
floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_promptrule_top_k",
budget_default=3,
asks="the operator's message, against rule triggers",
@@ -483,8 +497,8 @@ PRE_TOOL_RULE = RuleArm(
preference_slot=False,
tuning=Surface(
name="pre_tool_rule",
floor_key="kb_toolrule_threshold",
floor_default=0.68,
floor_key=RULE_FLOOR_KEY,
floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_toolrule_top_k",
budget_default=5,
asks="the command about to run, against rule triggers",
@@ -498,8 +512,8 @@ WRITE_PATH_RULE = RuleArm(
preference_slot=False,
tuning=Surface(
name="write_path_rule",
floor_key="kb_rulehint_threshold",
floor_default=0.72,
floor_key=RULE_FLOOR_KEY,
floor_default=RULE_FLOOR_DEFAULT,
budget_key="kb_rulehint_top_k",
budget_default=5,
asks="the code being written, against rule triggers",
@@ -110,6 +110,9 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
Derived rather than listed so a seventh surface is covered the moment it is
added here, which is the only way this stays true.
A floor key several surfaces share resolves to the first of them;
`set_dial` records the move on every surface in `sharing_floor`.
"""
for surface in SURFACES.values():
if key == surface.floor_key:
@@ -119,6 +122,16 @@ def dial_for_key(key: str) -> tuple[str, str] | None:
return None
def sharing_floor(name: str) -> list[str]:
"""Every surface that reads the same floor key as `name`, itself included.
The rule arms share one bar, so moving one surface's floor moves its
siblings, and each of them needs the event that says so.
"""
key = get_surface(name).floor_key
return [n for n, s in SURFACES.items() if s.floor_key == key]
async def floor_for(user_id: int, name: str) -> float:
"""This install's current floor for a surface, clamped to [0, 1]."""
s = get_surface(name)
+15 -6
View File
@@ -44,6 +44,7 @@ from grimoire.services.retrieval_surfaces import (
budget_for,
floor_for,
get_surface,
sharing_floor,
surface_names,
)
from grimoire.services.settings import set_setting
@@ -257,6 +258,10 @@ async def set_dial(
"reason": text, "actor": actor, "unchanged": True,
}
# A shared floor moves every surface that reads it, and each one's history
# has to show the move, or its telemetry changes with no event to explain it.
moved = sharing_floor(surface) if dial == "floor" else [surface]
await set_setting(user_id, key, stored)
async with async_session() as session:
# Stamped with the space this number was chosen in. Read at
@@ -264,12 +269,13 @@ async def set_dial(
# the caller recording it are the same call, so there is no window in
# which they could disagree.
stamp = calibration_stamp()
session.add(RetrievalTuningEvent(
user_id=user_id, surface=surface, dial=dial,
old_value=old, new_value=applied, actor=actor, reason=text,
embedding_model=stamp["embedding_model"],
shape_version=stamp["shape_version"],
))
for name in moved:
session.add(RetrievalTuningEvent(
user_id=user_id, surface=name, dial=dial,
old_value=old, new_value=applied, actor=actor, reason=text,
embedding_model=stamp["embedding_model"],
shape_version=stamp["shape_version"],
))
await session.commit()
return {
@@ -286,6 +292,9 @@ async def set_dial(
# Always present, both ways round: a caller that has to test for the
# key's absence to learn the answer will eventually forget to.
"unchanged": False,
# The other surfaces this move carried with it: a shared floor is one
# number, and a caller tuning one rule arm has tuned them all.
"moved_with": [n for n in moved if n != surface],
}