refactor(retrieval): one registry for every surface's floor and budget (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / Python tests (push) Failing after 1m3s
CI & Build / Build & push image (push) Skipped

Groundwork for the step's real change. The operator's decision is that the
floor is chosen and adjusted by the model using it, not shipped as a value
somebody has to defend:

  "we need a model consistent surface for the adjustment of these floor values.
   the user should be able to touch it but the model should be the thing
   handling it 9 times out of 10."

A tuning surface cannot be consistent across six arms that each spell their
configuration differently, so the arms stop owning their numbers.
`retrieval_surfaces.SURFACES` names each one, its floor key and default, its
budget key and default, and — because they are rendered by the tuning tool and
the Settings UI — what it asks, over what corpus, and how often it fires. A
floor cannot be moved responsibly by anyone who does not know those three.

Three things fall out:

- **`k` becomes a real budget everywhere.** Only auto-inject had a configurable
  one; `RULEHINT_LIMIT`, `PROMPTRULE_LIMIT` and `reply_preferences.LIMIT` were
  constants. `k` is what binds under a low floor, so it has to be settable per
  surface — and per surface is the point, since `pre_tool_rule` fires before
  every Bash call while `prompt_rule` fires once a turn.
- **`write_path` gets its own budget, inherited not reset.** It shared
  auto-inject's outright on the argument that "how many titles at once" means
  the same thing on both. It does not, for the same reason. Unset, it still
  reads auto-inject's key, so an install that tuned the shared knob does not
  silently drop to a new default.
- **The duplicated read-and-clamp goes.** That shape is canon #2860 across 295
  of 372 judged siblings. Survivable while the numbers were constants; not once
  they are meant to move.

The long measurement comments stay exactly where they are — #2223's noise-floor
probe, #3853's command-vs-code split, #3851's band measurement. The constants
they annotate now alias the registry, so there is one value and the reasoning
still sits beside it.

Tests build the write-path config from the registry (`helpers.writepath_cfg`)
instead of from hand-written dicts. That is not tidiness: the rule arms read
their numbers inside a fail-open `except`, so a dict missing one key does not
raise where a reader would see it — the arm silently becomes a no-op that reads
exactly like "fired and found nothing". Ten hand-written dicts each looked
complete on the day they were typed.

tests/test_retrieval_surfaces.py pins the identity everything rests on: a
surface's name IS its telemetry source. Nothing in the type system says so —
`record_retrieval(source="pre_tool_rule")` is a literal in another file — and
renaming one without the other yields an arm that can be tuned and not
measured, or measured and not tuned, with no symptom either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-17 11:17:23 -04:00
co-authored by Claude Opus 5
parent 5d47342fbc
commit 09b48457ff
10 changed files with 644 additions and 172 deletions
+82 -85
View File
@@ -31,6 +31,12 @@ from scribe.services.embeddings import semantic_search_notes, semantic_search_ru
from scribe.services.note_usage import record_surfaced
from scribe.services.rule_usage import record_rule_surfaced
from scribe.services.supersession import superseded_ids
from scribe.services.retrieval_surfaces import (
MAX_BUDGET,
SURFACES,
budget_for,
floor_for,
)
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.settings import get_setting
@@ -54,12 +60,16 @@ _GOAL_CHARS = 200
# and are meant to be tuned from retrieval_logs (source='auto_inject') once data
# accrues — they're exposed in the Settings UI, no restart needed.
AUTOINJECT_ENABLED_KEY = "kb_autoinject_enabled"
AUTOINJECT_THRESHOLD_KEY = "kb_autoinject_threshold"
AUTOINJECT_TOP_K_KEY = "kb_autoinject_top_k"
# The key and the value both live in the registry now (#4102); these names
# survive because the comments above them are where each number's measurement
# is recorded, and because the Settings UI agreement test pairs them with the
# Vue refs by name. One definition, two readable places.
AUTOINJECT_THRESHOLD_KEY = SURFACES["auto_inject"].floor_key
AUTOINJECT_TOP_K_KEY = SURFACES["auto_inject"].budget_key
AUTOINJECT_DEFAULT_ENABLED = True
AUTOINJECT_DEFAULT_THRESHOLD = 0.55
AUTOINJECT_DEFAULT_TOP_K = 3
AUTOINJECT_DEFAULT_THRESHOLD = SURFACES["auto_inject"].floor_default
AUTOINJECT_DEFAULT_TOP_K = SURFACES["auto_inject"].budget_default
# The write-path trigger (#2082) gets its own on/off switch, its own threshold,
# and shares only top-k. It originally shared the threshold too, on the argument
@@ -83,9 +93,9 @@ AUTOINJECT_DEFAULT_TOP_K = 3
# pull-through (#2085) once a real corpus accrues; a cross-encoder rerank
# (#1038) would subsume this bump.
WRITEPATH_ENABLED_KEY = "kb_writepath_enabled"
WRITEPATH_THRESHOLD_KEY = "kb_writepath_threshold"
WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
WRITEPATH_DEFAULT_ENABLED = True
WRITEPATH_DEFAULT_THRESHOLD = 0.68
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
# The standing-rule arm (milestone 307) gets its own bar — the split #2223 made
# one surface down, now made for the THIRD corpus. It inherited 0.68 above, and
@@ -133,8 +143,8 @@ WRITEPATH_DEFAULT_THRESHOLD = 0.68
# case 0.72 was calibrated on, and the telemetry says it is working — the
# write-path rule arm speaks on 37% of its calls and its refused mass sits at
# p50 0.6989, comfortably under the bar rather than piled against it.
RULEHINT_THRESHOLD_KEY = "kb_rulehint_threshold"
RULEHINT_DEFAULT_THRESHOLD = 0.72
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S (#3853).
#
@@ -193,8 +203,8 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
# eligible. Retrieval is ownership-scoped, not project-scoped. Scoping it
# would drop that ceiling and widen the 0.0115, which is the larger fix and
# the reason to settle project scoping before tuning this number twice.
TOOLRULE_THRESHOLD_KEY = "kb_toolrule_threshold"
TOOLRULE_DEFAULT_THRESHOLD = 0.68
TOOLRULE_THRESHOLD_KEY = SURFACES["pre_tool_rule"].floor_key
TOOLRULE_DEFAULT_THRESHOLD = SURFACES["pre_tool_rule"].floor_default
# A SET OF RULES PER ACT, NOT THE SINGLE BEST ONE (#3851).
#
@@ -223,7 +233,10 @@ TOOLRULE_DEFAULT_THRESHOLD = 0.68
# and nothing else, so a moment with one clearly-relevant rule still shows
# one, and a moment with four shows four. The corpus decides, not a constant.
# The cap survives as a ceiling on the worst case, not as the usual answer.
RULEHINT_LIMIT = 5
# NOW A DEFAULT RATHER THAN A CAP (#4102): both rule act-arms read their own
# budget from the registry, so this is the value an install starts at and not
# the value it is stuck with. The reasoning below is why 5 is where it starts.
RULEHINT_LIMIT = SURFACES["write_path_rule"].budget_default
# MEASURED, NOT REASONED — and the reasoning it replaced was wrong (#3851).
#
@@ -433,7 +446,11 @@ _CONCEPT_MIN_CHARS = 16
_AUTOINJECT_BAND = 0.10
# Hard ceiling on top-k regardless of the user's setting — this is an
# awareness menu (titles only), never a content dump.
_AUTOINJECT_MAX_TOP_K = 10
# The budget ceiling, now shared by every surface rather than owned by this
# one (#4102). It bounds the same thing everywhere — how many lines a single
# unsolicited injection may occupy — so one surface having a private ceiling
# was an accident of which arm got a configurable budget first.
_AUTOINJECT_MAX_TOP_K = MAX_BUDGET
# --- the prompt-boundary rule arm (#3852) ------------------------------------
#
@@ -446,7 +463,7 @@ _AUTOINJECT_MAX_TOP_K = 10
# The operator's message is the only query that exists before one is composed,
# and this arm is what runs against it. Until now that hook searched notes
# alone, so no rule had ever been retrieved against a thing the operator said.
PROMPTRULE_THRESHOLD_KEY = "kb_promptrule_threshold"
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
# INHERITED FROM THE ACT ARMS, AND NOT YET EARNED HERE. 0.72 was tuned against
# code and shell commands. An operator's prose is a different query shape
# against the same documents, and nothing yet says the two distributions line
@@ -458,7 +475,7 @@ PROMPTRULE_THRESHOLD_KEY = "kb_promptrule_threshold"
# front of a corpus that binds. Every call is logged under `prompt_rule` from
# the first deploy, so a few days of real traffic settles it — read
# `near_miss_samples` (#3807) before moving this, not the percentile alone.
PROMPTRULE_DEFAULT_THRESHOLD = 0.72
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
# MORE THAN THE ACT ARMS' SINGLE SLOT, anchored on this hook's budget rather
# than theirs. RULEHINT_LIMIT is 1 because that arm fires before EVERY Bash
@@ -469,7 +486,7 @@ PROMPTRULE_DEFAULT_THRESHOLD = 0.72
# And a prompt genuinely contains more than one act. "Merge to main and then
# start on X" is two, governed by different rules; k=1 cannot serve that case
# at all, where the act arms never face it because a command is one thing.
PROMPTRULE_LIMIT = 3
PROMPTRULE_LIMIT = SURFACES["prompt_rule"].budget_default
# THE COMPLETION-REPORT ARM'S OWN BAR (services/reply_preferences.py).
#
@@ -487,8 +504,8 @@ PROMPTRULE_LIMIT = 3
# Kept at the prose arm's starting value rather than tuned: the split is what
# makes the two independently movable, and a default is a product decision
# that this install's corpus cannot settle (rule 115).
REPORTPREF_THRESHOLD_KEY = "kb_reportpref_threshold"
REPORTPREF_DEFAULT_THRESHOLD = 0.72
REPORTPREF_THRESHOLD_KEY = SURFACES["report_preference"].floor_key
REPORTPREF_DEFAULT_THRESHOLD = SURFACES["report_preference"].floor_default
def _slugify(text: str) -> str:
@@ -592,8 +609,18 @@ async def build_process_manifest(user_id: int) -> dict:
async def get_autoinject_config(user_id: int) -> dict:
"""Resolve a user's auto-inject settings, falling back to the defaults.
Returns {"enabled": bool, "threshold": float, "top_k": int}, clamped to
sane ranges (threshold to [0,1]; top_k to [1, _AUTOINJECT_MAX_TOP_K]).
Returns {"enabled": bool, "threshold": float, "top_k": int}.
THE TWO NUMBERS COME FROM THE REGISTRY NOW (#4102). They used to be read and
clamped here, and identically again in `get_writepath_config`, and again in
three rule arms, and once more in `reply_preferences` — the duplication the
shape ledger counts as canon #2860 across 295 of 372 siblings. That was
tolerable while the values were shipped constants. It stops being tolerable
once a tool is expected to MOVE them, because a tuning surface cannot be
consistent across arms that each spell their configuration differently.
`enabled` stays here: it is this surface's own switch, not a tunable number,
and the registry deliberately holds only the pair a floor-tuner touches.
"""
enabled_raw = await get_setting(
user_id, AUTOINJECT_ENABLED_KEY,
@@ -601,21 +628,11 @@ async def get_autoinject_config(user_id: int) -> dict:
)
enabled = enabled_raw.strip().lower() in ("true", "1", "yes", "on")
try:
threshold = float(await get_setting(
user_id, AUTOINJECT_THRESHOLD_KEY, str(AUTOINJECT_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
threshold = AUTOINJECT_DEFAULT_THRESHOLD
threshold = min(1.0, max(0.0, threshold))
try:
top_k = int(float(await get_setting(
user_id, AUTOINJECT_TOP_K_KEY, str(AUTOINJECT_DEFAULT_TOP_K))))
except (TypeError, ValueError):
top_k = AUTOINJECT_DEFAULT_TOP_K
top_k = min(_AUTOINJECT_MAX_TOP_K, max(1, top_k))
return {"enabled": enabled, "threshold": threshold, "top_k": top_k}
return {
"enabled": enabled,
"threshold": await floor_for(user_id, "auto_inject"),
"top_k": await budget_for(user_id, "auto_inject"),
}
def _record_kind(note) -> str:
@@ -1024,13 +1041,8 @@ async def build_prompt_rule_hint(
return out
try:
try:
threshold = float(await get_setting(
user_id, PROMPTRULE_THRESHOLD_KEY,
str(PROMPTRULE_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
threshold = PROMPTRULE_DEFAULT_THRESHOLD
threshold = min(1.0, max(0.0, threshold))
threshold = await floor_for(user_id, "prompt_rule")
limit = await budget_for(user_id, "prompt_rule")
t0 = time.perf_counter()
_rep: dict = {}
@@ -1041,7 +1053,7 @@ async def build_prompt_rule_hint(
# sessions — this surface speaks unasked, and a whole-rulebook answer
# is only right for someone who asked the whole rulebook.
hits = await semantic_search_rules(
user_id, q, limit=PROMPTRULE_LIMIT, threshold=threshold,
user_id, q, limit=limit, threshold=threshold,
report=_rep, project_id=project_id or None,
)
duration_ms = (time.perf_counter() - t0) * 1000.0
@@ -1057,7 +1069,7 @@ async def build_prompt_rule_hint(
# and unverified for this corpus, so the zero rows are the point.
record_retrieval(
user_id=user_id, source="prompt_rule", query=q,
threshold=threshold, limit=PROMPTRULE_LIMIT,
threshold=threshold, limit=limit,
project_id=project_id,
is_task=None, results=fresh, duration_ms=duration_ms,
best_available=_rep.get("best_available_score"),
@@ -1294,52 +1306,37 @@ def concept_query(code: str) -> str:
async def get_writepath_config(user_id: int) -> dict:
"""Write-path trigger settings: its own `enabled` and `threshold`, auto-inject's top_k.
"""Write-path trigger settings and the two rule arms' numbers alongside.
The threshold OVERRIDES the inherited auto-inject value — code embeddings
have a much higher similarity floor than prose, so the two surfaces need
different bars. See WRITEPATH_DEFAULT_THRESHOLD for the measurements (#2223).
top_k is still shared: "how many titles at once" means the same thing on
both surfaces, and nothing suggests they want different ceilings.
Three surfaces' worth of configuration arrives in one call because one hook
request drives all three arms. They are still three SURFACES with three
independent pairs, resolved from the registry (#4102).
The floors were split apart one at a time, each on its own measurement, and
those measurements are recorded where the defaults are: WRITEPATH (#2223 —
code embeddings sit on a much higher similarity floor than prose), RULEHINT
(a third corpus again), TOOLRULE (#3853 — a shell command is a different
query shape from a code payload and scores lower for the same relevance).
THE BUDGET IS NOW SPLIT TOO. `top_k` used to be auto-inject's outright, on
the argument that "how many titles at once" means the same thing on both
surfaces. It does not: this arm fires before every Write and Edit while
auto-inject fires once a turn, so the same number buys wildly different
amounts of attention. `write_path` inherits auto-inject's value when it has
none of its own, so no install that tuned the shared knob loses it.
"""
cfg = await get_autoinject_config(user_id)
enabled_raw = await get_setting(
user_id, WRITEPATH_ENABLED_KEY,
"true" if WRITEPATH_DEFAULT_ENABLED else "false",
)
try:
threshold = float(await get_setting(
user_id, WRITEPATH_THRESHOLD_KEY, str(WRITEPATH_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
threshold = WRITEPATH_DEFAULT_THRESHOLD
threshold = min(1.0, max(0.0, threshold))
try:
rule_threshold = float(await get_setting(
user_id, RULEHINT_THRESHOLD_KEY, str(RULEHINT_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
rule_threshold = RULEHINT_DEFAULT_THRESHOLD
rule_threshold = min(1.0, max(0.0, rule_threshold))
try:
tool_rule_threshold = float(await get_setting(
user_id, TOOLRULE_THRESHOLD_KEY, str(TOOLRULE_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
tool_rule_threshold = TOOLRULE_DEFAULT_THRESHOLD
tool_rule_threshold = min(1.0, max(0.0, tool_rule_threshold))
return {
**cfg,
"enabled": enabled_raw.strip().lower() in ("true", "1", "yes", "on"),
"threshold": threshold,
# Its own bar, for a third corpus — see RULEHINT_DEFAULT_THRESHOLD.
"rule_threshold": rule_threshold,
# And the COMMAND arm's own bar again, for the same reason one level
# down: a shell command is a different query shape from a code payload
# and scores lower for the same relevance (#3853). Separate keys, so an
# install can move one without the other — which is the whole finding.
"tool_rule_threshold": tool_rule_threshold,
"threshold": await floor_for(user_id, "write_path"),
"top_k": await budget_for(user_id, "write_path"),
"rule_threshold": await floor_for(user_id, "write_path_rule"),
"rule_top_k": await budget_for(user_id, "write_path_rule"),
"tool_rule_threshold": await floor_for(user_id, "pre_tool_rule"),
"tool_rule_top_k": await budget_for(user_id, "pre_tool_rule"),
}
def _rule_band(hits: list) -> list:
@@ -1988,7 +1985,7 @@ async def build_write_path_hint(
rule_t0 = time.perf_counter()
_rep_wpr: dict = {}
hits = await semantic_search_rules(
user_id, code or path, limit=RULEHINT_LIMIT,
user_id, code or path, limit=cfg["rule_top_k"],
threshold=cfg["rule_threshold"],
report=_rep_wpr, project_id=project_id or None,
)
@@ -2049,7 +2046,7 @@ async def build_write_path_hint(
# the same readout.
record_retrieval(
user_id=user_id, source="write_path_rule", query=code or path,
threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
threshold=cfg["rule_threshold"], limit=cfg["rule_top_k"],
project_id=project_id,
is_task=None, results=fresh, duration_ms=rule_ms,
best_available=_rep_wpr.get("best_available_score"),
@@ -2145,7 +2142,7 @@ async def build_tool_rule_hint(
t0 = time.perf_counter()
_rep_ptr: dict = {}
hits = await semantic_search_rules(
user_id, query, limit=RULEHINT_LIMIT,
user_id, query, limit=cfg["tool_rule_top_k"],
threshold=cfg["tool_rule_threshold"],
report=_rep_ptr, project_id=project_id or None,
)
@@ -2168,7 +2165,7 @@ async def build_tool_rule_hint(
# failure the arm was built to stop.
record_retrieval(
user_id=user_id, source="pre_tool_rule", query=query,
threshold=cfg["tool_rule_threshold"], limit=RULEHINT_LIMIT,
threshold=cfg["tool_rule_threshold"], limit=cfg["tool_rule_top_k"],
project_id=project_id,
is_task=None, results=fresh, duration_ms=duration_ms,
best_available=_rep_ptr.get("best_available_score"),
+15 -14
View File
@@ -61,13 +61,9 @@ import logging
import time
from scribe.services.embeddings import semantic_search_rules
from scribe.services.plugin_context import (
REPORTPREF_DEFAULT_THRESHOLD,
REPORTPREF_THRESHOLD_KEY,
)
from scribe.services.retrieval_surfaces import SURFACES, budget_for, floor_for
from scribe.services.retrieval_telemetry import record_retrieval
from scribe.services.rule_usage import record_rule_surfaced
from scribe.services.settings import get_setting
logger = logging.getLogger(__name__)
@@ -85,16 +81,20 @@ COMPLETION_QUERY = (
# A handful, not a menu. More than a few shape preferences for ONE kind of
# reply would contradict each other before they helped; the limit is here to
# keep one noisy corpus from turning a status change into a wall of text.
LIMIT = 3
# The STARTING budget, not the budget (#4102). Both numbers this arm runs on
# now come from the surface registry, so the model that reads this arm's
# telemetry can move either — which matters more here than anywhere else,
# because a fixed query makes this arm's score a constant and a floor a hair
# above it produces a dead arm no amount of traffic will ever reveal.
LIMIT = SURFACES["report_preference"].budget_default
async def _threshold(user_id: int) -> float:
try:
value = float(await get_setting(
user_id, REPORTPREF_THRESHOLD_KEY, str(REPORTPREF_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
value = REPORTPREF_DEFAULT_THRESHOLD
return min(1.0, max(0.0, value))
return await floor_for(user_id, SOURCE)
async def _limit(user_id: int) -> int:
return await budget_for(user_id, SOURCE)
async def completion_preferences(user_id: int, *, project_id: int | None = None) -> list[dict]:
@@ -115,16 +115,17 @@ async def completion_preferences(user_id: int, *, project_id: int | None = None)
"""
try:
threshold = await _threshold(user_id)
limit = await _limit(user_id)
report: dict = {}
t0 = time.perf_counter()
hits = await semantic_search_rules(
user_id, COMPLETION_QUERY, limit=LIMIT, threshold=threshold,
user_id, COMPLETION_QUERY, limit=limit, threshold=threshold,
kind="preference", report=report, project_id=project_id,
)
hits = [(score, rule) for score, rule in hits if rule.kind == "preference"]
record_retrieval(
user_id=user_id, source=SOURCE, query=COMPLETION_QUERY,
threshold=threshold, limit=LIMIT, project_id=project_id,
threshold=threshold, limit=limit, project_id=project_id,
is_task=None, results=hits,
best_available=report.get("best_available_score"),
best_available_id=report.get("best_available_id"),
+243
View File
@@ -0,0 +1,243 @@
"""One registry of the retrieval surfaces, and the two numbers each one has (#4102).
WHY THIS EXISTS
Six push arms each carried their own loose copy of the same shape: a settings
key, a default, and a limit that was usually a module constant nobody could
change. `_threshold()` was duplicated so widely that the shape ledger counts it
as canon across 295 of 372 judged siblings (snippet #2860). That was survivable
while the numbers were shipped constants an operator occasionally edited.
It stops being survivable once the numbers are meant to MOVE. The operator's
decision for this step:
"the floor should be chosen and adjusted by the model using it. we've come
back to something either fails or has to be looked at by the user we need a
model consistent surface for the adjustment of these floor values. the user
should be able to touch it but the model should be the thing handling it 9
times out of 10."
A tuning surface cannot be "model consistent" if every arm spells its own
configuration differently. So the arms stop owning their numbers and read them
from here instead, and the tuning tool, the routes and the Settings UI all
enumerate THIS table rather than hard-coding six special cases.
WHAT A FLOOR IS NOW, AND WHAT IT IS NOT
It is not a relevance judgement. Relevance is decided by the reader, which is
the only participant that can read a trigger against a situation — that is the
milestone's whole argument, and the injected line has always said so out loud
("read it before deciding it does not apply").
A floor answers the cheaper question: *is this worth ranking at all*. `k` is
what binds, and `k` is a BUDGET — how much of this surface's attention a
candidate list may spend. An arm that fires before every Bash call cannot
afford what an arm that fires once a turn can.
WHY THE DEFAULTS BELOW ARE STARTING POINTS AND NOT ANSWERS
A cosine score is a distance in BAAI/bge-small-en-v1.5's vector space, measured
against THIS corpus. It cannot transfer to an install with different records,
and rule 115 forbids defending a shipped default from this instance's
telemetry — which is what made the old design unbuildable: every number was a
guess everywhere except here, and there was no mechanism that could ever
improve it.
The mechanism is the fix. Scribe ships a starting point and the means to
correct it, so the values below carry the measurement that motivated them (see
the long comments in `plugin_context.py`, which are kept where they are because
they record how each number was first arrived at) without claiming to be right
for anybody else.
HOW A FLOOR SHOULD ACTUALLY BE MOVED
By reading the records the floor refused — `retrieval_telemetry(
near_miss_samples=N)` returns them by id — and never by the percentile alone.
That is not a style preference; it is the one case where the two disagreed and
was checked. `report_preference` logged 69 consecutive declines with the
refused score 0.0006 under the bar, and every percentile said "lower it".
Reading the refused record showed it was rule 77 "Extract intent from loose
phrasing", a false positive, so lowering the bar would have delivered that rule
on every completion report ever written. The statistic and the correct action
pointed in opposite directions, and only opening the record could tell.
"""
from __future__ import annotations
from dataclasses import dataclass
from scribe.services.settings import get_setting
# A budget nobody should be able to set past. Not a tuning value — a guard on
# the worst case, so a mistyped setting cannot turn a menu into a wall of text.
# Shared by every surface because it bounds the same thing everywhere: how many
# lines a single unsolicited injection may occupy.
MAX_BUDGET = 10
@dataclass(frozen=True)
class Surface:
"""One push arm's tunable pair, plus enough prose to tune it responsibly.
`asks` / `over` / `fires` are not documentation for this file — they are
rendered by the tuning tool and the Settings UI. A floor cannot be moved
sensibly by anyone, model or human, who does not know what the query is, what
corpus it runs against, or how often it costs something. Those three facts
are exactly what separates these arms from each other, and they were
previously recoverable only by reading `plugin_context.py`.
"""
name: str
"""The telemetry `source` value, and the join key.
MUST equal the string this arm passes to `record_retrieval`. Everything
useful about tuning depends on that identity: the tool that moves a floor
and the table that says what the floor did have to be talking about the same
arm. A test asserts it rather than a comment asking nicely.
"""
floor_key: str
floor_default: float
budget_key: str
budget_default: int
asks: str
over: str
fires: str
budget_falls_back_to: str = ""
"""A budget key to inherit when this surface has none of its own set.
Only `write_path` uses it, and only because it USED to share auto-inject's
`top_k` outright. Giving it a key without this would silently reset the
budget of every install that had tuned the shared one — a behaviour change
delivered as a default, which is the shape of regression nobody reports
because nothing looks broken.
"""
SURFACES: dict[str, Surface] = {
"auto_inject": Surface(
name="auto_inject",
floor_key="kb_autoinject_threshold",
floor_default=0.55,
budget_key="kb_autoinject_top_k",
budget_default=3,
asks="the operator's message, as they typed it",
over="notes, snippets, processes and issues",
fires="once per operator turn",
),
"write_path": Surface(
name="write_path",
floor_key="kb_writepath_threshold",
floor_default=0.68,
budget_key="kb_writepath_top_k",
budget_default=3,
budget_falls_back_to="kb_autoinject_top_k",
asks="the code being written, rewritten as a concept query",
over="snippets and recorded issues",
fires="before every Write and Edit",
),
"write_path_rule": Surface(
name="write_path_rule",
floor_key="kb_rulehint_threshold",
floor_default=0.72,
budget_key="kb_rulehint_top_k",
budget_default=5,
asks="the code being written, against rule triggers",
over="global rules plus the bound project's own",
fires="before every Write and Edit",
),
"pre_tool_rule": Surface(
name="pre_tool_rule",
floor_key="kb_toolrule_threshold",
floor_default=0.68,
budget_key="kb_toolrule_top_k",
budget_default=5,
asks="the command about to run, against rule triggers",
over="global rules plus the bound project's own",
fires="before every Bash call — the busiest arm there is",
),
"prompt_rule": Surface(
name="prompt_rule",
floor_key="kb_promptrule_threshold",
floor_default=0.72,
budget_key="kb_promptrule_top_k",
budget_default=3,
asks="the operator's message, against rule triggers",
over="global rules plus the bound project's own",
fires="once per operator turn",
),
"report_preference": Surface(
name="report_preference",
floor_key="kb_reportpref_threshold",
floor_default=0.72,
budget_key="kb_reportpref_top_k",
budget_default=3,
# THE ONE FIXED QUERY, and the reason this arm behaves unlike the rest.
# The others score something that varies per call; this one scores a
# constant string, so its top score for a given corpus is also a
# constant. A floor a hair above that constant is not a quiet arm, it is
# a dead one, and no amount of traffic will ever reveal it — which is
# precisely how this arm spent 69 calls declining the same record.
asks="a fixed question about how to lay out a completion report",
over="preferences",
fires="when a task finishes",
),
}
# Reserved slots are deliberately absent. `preference_slot` and `reuse_slot`
# borrow their parent arm's floor and are hard-limited to one hit each, because
# their entire purpose is to guarantee a single line to a kind of record that
# keeps losing a general score contest (#2246, #3894). A budget of "1" is the
# feature; exposing it as tunable would invite setting it to 0 and silently
# removing the guarantee.
def surface_names() -> list[str]:
"""Every tunable surface, in a stable order for menus and listings."""
return list(SURFACES)
def get_surface(name: str) -> Surface:
"""Look one up, refusing an unknown name loudly.
A typo'd surface must not be writable. Settings keys are free-form strings
in a generic table, so a tuning call naming `pretool_rule` would otherwise
write a key nothing ever reads — a change that appears to succeed, reports a
new value, and alters nothing.
"""
try:
return SURFACES[name]
except KeyError:
raise ValueError(
f"unknown retrieval surface {name!r}. Tunable surfaces are: "
+ ", ".join(surface_names())
) from None
async def floor_for(user_id: int, name: str) -> float:
"""This install's current floor for a surface, clamped to [0, 1]."""
s = get_surface(name)
try:
value = float(await get_setting(user_id, s.floor_key, str(s.floor_default)))
except (TypeError, ValueError):
value = s.floor_default
return min(1.0, max(0.0, value))
async def budget_for(user_id: int, name: str) -> int:
"""This install's current budget for a surface, clamped to [1, MAX_BUDGET].
The lower clamp is 1, never 0: a surface turned off is turned off by its
`enabled` switch, which says so. A budget of zero would be an arm that runs
a search, logs a retrieval, and renders nothing — indistinguishable in the
telemetry from a bar nothing cleared, which is the exact confusion this
milestone exists to remove.
"""
s = get_surface(name)
raw = await get_setting(user_id, s.budget_key, "")
if not raw and s.budget_falls_back_to:
raw = await get_setting(user_id, s.budget_falls_back_to, "")
try:
value = int(float(raw)) if raw else s.budget_default
except (TypeError, ValueError):
value = s.budget_default
return min(MAX_BUDGET, max(1, value))