align(4909d): every rule arm that annotates reads one bar, kb_rule_threshold at 0.70
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m7s
CI & Build / Python tests (push) Failing after 1m30s
CI & Build / Build & push image (push) Skipped
prompt_rule, pre_tool_rule and write_path_rule now share one settings key, RULE_FLOOR_KEY = "kb_rule_threshold", default 0.70. The reply backstop keeps its own 0.80, because there the floor is a stop bar. Moving the shared floor through tune_retrieval or the Settings form records a tuning event on every surface that reads it (set_dial, via retrieval_surfaces.sharing_floor), and the result names the surfaces it carried as `moved_with`. Settings shows one "Standing-rule confidence threshold" control where it showed three. Operator ruling (2026-10-05): one shared bar for the rule arms, "especially since we're only taking the first x number of them" -- the budget limits what is shown; the bar only decides what is worth ranking. Kept a dial. Why 0.70, from records read rather than percentiles: - write_path_rule near-misses at 0.7199-0.72 were all false positives (rule 149 against a Lidarr comment, 157 against a retention Svelte file, 121 against an Archives restore, 164 against a config test, 131 against a Go package), so the old 0.72 was not hiding real hits -- but the band does noise control on this arm, so dropping 0.02 costs little. - prompt_rule: the operator set 0.70 by hand on 2026-09-18, after a 0.60 experiment admitted junk and was reverted. 0.70 is the number with a human decision behind it. - pre_tool_rule lines near 0.68 this session were mostly noise; rising to 0.70 trims them, and the band still narrows what clears. Retired keys: kb_promptrule_threshold, kb_toolrule_threshold and kb_rulehint_threshold are no longer read. Rows an install stored under them stay in the settings table, inert. On this instance the effective bars move pre_tool_rule 0.68 -> 0.70, write_path_rule 0.72 -> 0.70, prompt_rule stays 0.70; record_release_defaults writes those per-surface release rows on boot, so the telemetry shift has an event to explain it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -144,7 +144,7 @@ WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
|
||||
WRITEPATH_DEFAULT_ENABLED = True
|
||||
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
|
||||
|
||||
# The standing-rule arm gets its own bar, for a third corpus: 0.68 above is
|
||||
# The standing-rule arms get their own bar, for a third corpus: 0.68 above is
|
||||
# calibrated on code against note PROSE, not code against RULE TEXT.
|
||||
#
|
||||
# THE STRUCTURAL ARGUMENT, which holds on any install, including one with six
|
||||
@@ -155,64 +155,27 @@ WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
|
||||
# pool always returns something, so "the best match cleared the bar"
|
||||
# drifts from "a good match exists" toward "N things were ranked". A bar
|
||||
# calibrated for best-of-thousands is cleared by best-of-forty as
|
||||
# arithmetic rather than relevance. (The arms search every tier, which
|
||||
# makes the pool larger; the bar still errs toward silence on purpose.)
|
||||
# arithmetic rather than relevance.
|
||||
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS
|
||||
# corpus than note prose, and a more homogeneous corpus has a HIGHER
|
||||
# noise floor than prose's 0.55-0.63.
|
||||
#
|
||||
# WHY 0.72 AND NOT A NUMBER OFF A HISTOGRAM. The exact offset between prose's
|
||||
# floor and rule-text's is not derivable in general — it depends on how an
|
||||
# install writes its rules — so the default errs deliberately toward SILENCE
|
||||
# rather than toward recall, on an asymmetry that is itself structural: this
|
||||
# hint fires on EVERY write. A missed rule is recoverable, because the rule is
|
||||
# still in Grimoire and the agent can search it. A hint that cries wolf is not:
|
||||
# it teaches the reader to skip the whole block, and the surface is lost along
|
||||
# with the true positives it would have carried.
|
||||
# ONE BAR FOR EVERY RULE ARM THAT ANNOTATES. The write path, the command arm
|
||||
# and the prompt arm read the same key (`RULE_FLOOR_KEY` in retrieval_pipeline):
|
||||
# they search the same rules, and what limits each one is its budget and band.
|
||||
#
|
||||
# TUNE IT FROM YOUR OWN INSTANCE: `retrieval_telemetry` reports
|
||||
# `rule_usage.pull_through`. Raise this if rules arrive unread; lower it if
|
||||
# rules you needed never arrived. What would RETIRE it: a cross-encoder rerank,
|
||||
# which would make a similarity bar the wrong control entirely.
|
||||
#
|
||||
# This bar is the WRITE-PATH arm's. A code payload is long and rich, which is
|
||||
# the case 0.72 suits; the command arm has its own bar below.
|
||||
# `rule_usage.pull_through` and `near_miss_samples`. Raise it if rules arrive
|
||||
# unread; lower it if rules you needed never arrived. What would RETIRE it: a
|
||||
# cross-encoder rerank, which would make a similarity bar the wrong control
|
||||
# entirely.
|
||||
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
|
||||
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
|
||||
|
||||
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S.
|
||||
#
|
||||
# A write-path query is a code payload, long and rich, while a pre-tool query
|
||||
# is a shell command — often under a dozen words. Less text, less signal, lower
|
||||
# scores for the same relevance. At the write path's 0.72 the command arm
|
||||
# turns nearly everything away, with its refused mass piled just under the
|
||||
# line, which is the shape a bar set too high leaves.
|
||||
#
|
||||
# WHAT 0.68 IS CALIBRATED AGAINST. Replayed queries, consequential acts
|
||||
# against innocuous ones, scored against a rule corpus with symptom-bearing
|
||||
# triggers:
|
||||
#
|
||||
# 0.7571 git push origin dev consequential
|
||||
# 0.7245 cd ...; git fetch; git add -A consequential
|
||||
# 0.7193 git pull --rebase origin dev consequential
|
||||
# 0.6850 docker compose up -d consequential
|
||||
# ---------------------------------------- 0.68
|
||||
# 0.6735 wc -l src/*.py && date innocuous
|
||||
# 0.6544 grep -rn useState src/ innocuous
|
||||
# 0.6099 sed -n '120,160p' package.json innocuous
|
||||
# 0.6056 ls -la && cat README.md innocuous
|
||||
#
|
||||
# At 0.72 three of the four consequential acts would retrieve nothing, though
|
||||
# the right rules ranked first for each.
|
||||
#
|
||||
# THE SEPARATION IS NARROW (about 0.01), and that is a caveat, not a result.
|
||||
# Eight probes set a direction; they do not settle a number.
|
||||
# `near_miss_samples` on real traffic is what settles it, and this is the bar
|
||||
# to re-read first.
|
||||
#
|
||||
# WHY A LOWER BAR IS SAFE HERE. The band does noise control downstream: a marginal hit that
|
||||
# clears the bar still has to score within `_RULEHINT_BAND` of the top to be
|
||||
# rendered. The bar's job shrank, so the bar can.
|
||||
# THE COMMAND ARM READS THE SAME BAR. A shell command is short, so it scores
|
||||
# lower than a code payload for the same relevance and the shared bar admits
|
||||
# less here. The band does noise control downstream: a hit that clears the bar
|
||||
# still has to score within `_RULEHINT_BAND` of the top to be rendered.
|
||||
#
|
||||
# The innocuous scores near the line are largely CROSS-PROJECT BLEED rather
|
||||
# than bad ranking: another project's shell-command rule matching a shell
|
||||
@@ -478,17 +441,11 @@ def _autoinject_query(prompt: str, context: str) -> str:
|
||||
# The operator's message is the only query that exists before one is composed,
|
||||
# and this arm is what runs against it.
|
||||
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
|
||||
# INHERITED FROM THE ACT ARMS. 0.72 is tuned against
|
||||
# code and shell commands. An operator's prose is a different query shape
|
||||
# against the same documents, and nothing yet says the two distributions line
|
||||
# up — triggers are written in the vocabulary of the MOMENT, which for most
|
||||
# rules is act vocabulary, so prose may well score lower across the board.
|
||||
#
|
||||
# Starting at the act arms' number anyway is deliberate: it is the only value
|
||||
# with evidence behind it, and guessing lower would put an unmeasured bar in
|
||||
# front of a corpus that binds. Every call is logged under `prompt_rule`, so
|
||||
# real traffic settles it — read `near_miss_samples` before moving this, not
|
||||
# the percentile alone.
|
||||
# THE SAME BAR AS THE ACT ARMS. An operator's prose is a different query shape
|
||||
# against the same documents, and triggers are written in the vocabulary of
|
||||
# the moment, which for most rules is act vocabulary. Every call is logged
|
||||
# under `prompt_rule`, so read `near_miss_samples` before moving the shared
|
||||
# bar, not the percentile alone.
|
||||
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
|
||||
|
||||
# ANCHORED ON THIS HOOK'S BUDGET rather than the act arms'. This arm fires
|
||||
@@ -1171,9 +1128,8 @@ async def get_writepath_config(user_id: int) -> dict:
|
||||
independent pairs, resolved from the registry.
|
||||
|
||||
Each floor's reason is recorded where its default is: WRITEPATH (code
|
||||
embeddings sit on a much higher similarity floor than prose), RULEHINT (a
|
||||
third corpus again), TOOLRULE (a shell command is a different query shape
|
||||
from a code payload and scores lower for the same relevance).
|
||||
embeddings sit on a much higher similarity floor than prose) and RULEHINT
|
||||
(a third corpus again, whose one bar both rule arms read).
|
||||
|
||||
THE BUDGET IS SPLIT TOO. "How many titles at once" does not mean the same
|
||||
thing on both surfaces: this arm fires before every Write and Edit while
|
||||
|
||||
Reference in New Issue
Block a user