refactor(retrieval): one registry for every surface's floor and budget (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / Python tests (push) Failing after 1m3s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / Python tests (push) Failing after 1m3s
CI & Build / Build & push image (push) Skipped
Groundwork for the step's real change. The operator's decision is that the floor is chosen and adjusted by the model using it, not shipped as a value somebody has to defend: "we need a model consistent surface for the adjustment of these floor values. the user should be able to touch it but the model should be the thing handling it 9 times out of 10." A tuning surface cannot be consistent across six arms that each spell their configuration differently, so the arms stop owning their numbers. `retrieval_surfaces.SURFACES` names each one, its floor key and default, its budget key and default, and — because they are rendered by the tuning tool and the Settings UI — what it asks, over what corpus, and how often it fires. A floor cannot be moved responsibly by anyone who does not know those three. Three things fall out: - **`k` becomes a real budget everywhere.** Only auto-inject had a configurable one; `RULEHINT_LIMIT`, `PROMPTRULE_LIMIT` and `reply_preferences.LIMIT` were constants. `k` is what binds under a low floor, so it has to be settable per surface — and per surface is the point, since `pre_tool_rule` fires before every Bash call while `prompt_rule` fires once a turn. - **`write_path` gets its own budget, inherited not reset.** It shared auto-inject's outright on the argument that "how many titles at once" means the same thing on both. It does not, for the same reason. Unset, it still reads auto-inject's key, so an install that tuned the shared knob does not silently drop to a new default. - **The duplicated read-and-clamp goes.** That shape is canon #2860 across 295 of 372 judged siblings. Survivable while the numbers were constants; not once they are meant to move. The long measurement comments stay exactly where they are — #2223's noise-floor probe, #3853's command-vs-code split, #3851's band measurement. The constants they annotate now alias the registry, so there is one value and the reasoning still sits beside it. Tests build the write-path config from the registry (`helpers.writepath_cfg`) instead of from hand-written dicts. That is not tidiness: the rule arms read their numbers inside a fail-open `except`, so a dict missing one key does not raise where a reader would see it — the arm silently becomes a no-op that reads exactly like "fired and found nothing". Ten hand-written dicts each looked complete on the day they were typed. tests/test_retrieval_surfaces.py pins the identity everything rests on: a surface's name IS its telemetry source. Nothing in the type system says so — `record_retrieval(source="pre_tool_rule")` is a literal in another file — and renaming one without the other yields an arm that can be tuned and not measured, or measured and not tuned, with no symptom either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -31,6 +31,12 @@ from scribe.services.embeddings import semantic_search_notes, semantic_search_ru
|
||||
from scribe.services.note_usage import record_surfaced
|
||||
from scribe.services.rule_usage import record_rule_surfaced
|
||||
from scribe.services.supersession import superseded_ids
|
||||
from scribe.services.retrieval_surfaces import (
|
||||
MAX_BUDGET,
|
||||
SURFACES,
|
||||
budget_for,
|
||||
floor_for,
|
||||
)
|
||||
from scribe.services.retrieval_telemetry import record_retrieval
|
||||
from scribe.services.settings import get_setting
|
||||
|
||||
@@ -54,12 +60,16 @@ _GOAL_CHARS = 200
|
||||
# and are meant to be tuned from retrieval_logs (source='auto_inject') once data
|
||||
# accrues — they're exposed in the Settings UI, no restart needed.
|
||||
AUTOINJECT_ENABLED_KEY = "kb_autoinject_enabled"
|
||||
AUTOINJECT_THRESHOLD_KEY = "kb_autoinject_threshold"
|
||||
AUTOINJECT_TOP_K_KEY = "kb_autoinject_top_k"
|
||||
# The key and the value both live in the registry now (#4102); these names
|
||||
# survive because the comments above them are where each number's measurement
|
||||
# is recorded, and because the Settings UI agreement test pairs them with the
|
||||
# Vue refs by name. One definition, two readable places.
|
||||
AUTOINJECT_THRESHOLD_KEY = SURFACES["auto_inject"].floor_key
|
||||
AUTOINJECT_TOP_K_KEY = SURFACES["auto_inject"].budget_key
|
||||
|
||||
AUTOINJECT_DEFAULT_ENABLED = True
|
||||
AUTOINJECT_DEFAULT_THRESHOLD = 0.55
|
||||
AUTOINJECT_DEFAULT_TOP_K = 3
|
||||
AUTOINJECT_DEFAULT_THRESHOLD = SURFACES["auto_inject"].floor_default
|
||||
AUTOINJECT_DEFAULT_TOP_K = SURFACES["auto_inject"].budget_default
|
||||
|
||||
# The write-path trigger (#2082) gets its own on/off switch, its own threshold,
|
||||
# and shares only top-k. It originally shared the threshold too, on the argument
|
||||
@@ -83,9 +93,9 @@ AUTOINJECT_DEFAULT_TOP_K = 3
|
||||
# pull-through (#2085) once a real corpus accrues; a cross-encoder rerank
|
||||
# (#1038) would subsume this bump.
|
||||
WRITEPATH_ENABLED_KEY = "kb_writepath_enabled"
|
||||
WRITEPATH_THRESHOLD_KEY = "kb_writepath_threshold"
|
||||
WRITEPATH_THRESHOLD_KEY = SURFACES["write_path"].floor_key
|
||||
WRITEPATH_DEFAULT_ENABLED = True
|
||||
WRITEPATH_DEFAULT_THRESHOLD = 0.68
|
||||
WRITEPATH_DEFAULT_THRESHOLD = SURFACES["write_path"].floor_default
|
||||
|
||||
# The standing-rule arm (milestone 307) gets its own bar — the split #2223 made
|
||||
# one surface down, now made for the THIRD corpus. It inherited 0.68 above, and
|
||||
@@ -133,8 +143,8 @@ WRITEPATH_DEFAULT_THRESHOLD = 0.68
|
||||
# case 0.72 was calibrated on, and the telemetry says it is working — the
|
||||
# write-path rule arm speaks on 37% of its calls and its refused mass sits at
|
||||
# p50 0.6989, comfortably under the bar rather than piled against it.
|
||||
RULEHINT_THRESHOLD_KEY = "kb_rulehint_threshold"
|
||||
RULEHINT_DEFAULT_THRESHOLD = 0.72
|
||||
RULEHINT_THRESHOLD_KEY = SURFACES["write_path_rule"].floor_key
|
||||
RULEHINT_DEFAULT_THRESHOLD = SURFACES["write_path_rule"].floor_default
|
||||
|
||||
# THE COMMAND ARM'S OWN BAR, AND WHY IT IS NOT THE WRITE PATH'S (#3853).
|
||||
#
|
||||
@@ -193,8 +203,8 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
|
||||
# eligible. Retrieval is ownership-scoped, not project-scoped. Scoping it
|
||||
# would drop that ceiling and widen the 0.0115, which is the larger fix and
|
||||
# the reason to settle project scoping before tuning this number twice.
|
||||
TOOLRULE_THRESHOLD_KEY = "kb_toolrule_threshold"
|
||||
TOOLRULE_DEFAULT_THRESHOLD = 0.68
|
||||
TOOLRULE_THRESHOLD_KEY = SURFACES["pre_tool_rule"].floor_key
|
||||
TOOLRULE_DEFAULT_THRESHOLD = SURFACES["pre_tool_rule"].floor_default
|
||||
|
||||
# A SET OF RULES PER ACT, NOT THE SINGLE BEST ONE (#3851).
|
||||
#
|
||||
@@ -223,7 +233,10 @@ TOOLRULE_DEFAULT_THRESHOLD = 0.68
|
||||
# and nothing else, so a moment with one clearly-relevant rule still shows
|
||||
# one, and a moment with four shows four. The corpus decides, not a constant.
|
||||
# The cap survives as a ceiling on the worst case, not as the usual answer.
|
||||
RULEHINT_LIMIT = 5
|
||||
# NOW A DEFAULT RATHER THAN A CAP (#4102): both rule act-arms read their own
|
||||
# budget from the registry, so this is the value an install starts at and not
|
||||
# the value it is stuck with. The reasoning below is why 5 is where it starts.
|
||||
RULEHINT_LIMIT = SURFACES["write_path_rule"].budget_default
|
||||
|
||||
# MEASURED, NOT REASONED — and the reasoning it replaced was wrong (#3851).
|
||||
#
|
||||
@@ -433,7 +446,11 @@ _CONCEPT_MIN_CHARS = 16
|
||||
_AUTOINJECT_BAND = 0.10
|
||||
# Hard ceiling on top-k regardless of the user's setting — this is an
|
||||
# awareness menu (titles only), never a content dump.
|
||||
_AUTOINJECT_MAX_TOP_K = 10
|
||||
# The budget ceiling, now shared by every surface rather than owned by this
|
||||
# one (#4102). It bounds the same thing everywhere — how many lines a single
|
||||
# unsolicited injection may occupy — so one surface having a private ceiling
|
||||
# was an accident of which arm got a configurable budget first.
|
||||
_AUTOINJECT_MAX_TOP_K = MAX_BUDGET
|
||||
|
||||
# --- the prompt-boundary rule arm (#3852) ------------------------------------
|
||||
#
|
||||
@@ -446,7 +463,7 @@ _AUTOINJECT_MAX_TOP_K = 10
|
||||
# The operator's message is the only query that exists before one is composed,
|
||||
# and this arm is what runs against it. Until now that hook searched notes
|
||||
# alone, so no rule had ever been retrieved against a thing the operator said.
|
||||
PROMPTRULE_THRESHOLD_KEY = "kb_promptrule_threshold"
|
||||
PROMPTRULE_THRESHOLD_KEY = SURFACES["prompt_rule"].floor_key
|
||||
# INHERITED FROM THE ACT ARMS, AND NOT YET EARNED HERE. 0.72 was tuned against
|
||||
# code and shell commands. An operator's prose is a different query shape
|
||||
# against the same documents, and nothing yet says the two distributions line
|
||||
@@ -458,7 +475,7 @@ PROMPTRULE_THRESHOLD_KEY = "kb_promptrule_threshold"
|
||||
# front of a corpus that binds. Every call is logged under `prompt_rule` from
|
||||
# the first deploy, so a few days of real traffic settles it — read
|
||||
# `near_miss_samples` (#3807) before moving this, not the percentile alone.
|
||||
PROMPTRULE_DEFAULT_THRESHOLD = 0.72
|
||||
PROMPTRULE_DEFAULT_THRESHOLD = SURFACES["prompt_rule"].floor_default
|
||||
|
||||
# MORE THAN THE ACT ARMS' SINGLE SLOT, anchored on this hook's budget rather
|
||||
# than theirs. RULEHINT_LIMIT is 1 because that arm fires before EVERY Bash
|
||||
@@ -469,7 +486,7 @@ PROMPTRULE_DEFAULT_THRESHOLD = 0.72
|
||||
# And a prompt genuinely contains more than one act. "Merge to main and then
|
||||
# start on X" is two, governed by different rules; k=1 cannot serve that case
|
||||
# at all, where the act arms never face it because a command is one thing.
|
||||
PROMPTRULE_LIMIT = 3
|
||||
PROMPTRULE_LIMIT = SURFACES["prompt_rule"].budget_default
|
||||
|
||||
# THE COMPLETION-REPORT ARM'S OWN BAR (services/reply_preferences.py).
|
||||
#
|
||||
@@ -487,8 +504,8 @@ PROMPTRULE_LIMIT = 3
|
||||
# Kept at the prose arm's starting value rather than tuned: the split is what
|
||||
# makes the two independently movable, and a default is a product decision
|
||||
# that this install's corpus cannot settle (rule 115).
|
||||
REPORTPREF_THRESHOLD_KEY = "kb_reportpref_threshold"
|
||||
REPORTPREF_DEFAULT_THRESHOLD = 0.72
|
||||
REPORTPREF_THRESHOLD_KEY = SURFACES["report_preference"].floor_key
|
||||
REPORTPREF_DEFAULT_THRESHOLD = SURFACES["report_preference"].floor_default
|
||||
|
||||
|
||||
def _slugify(text: str) -> str:
|
||||
@@ -592,8 +609,18 @@ async def build_process_manifest(user_id: int) -> dict:
|
||||
async def get_autoinject_config(user_id: int) -> dict:
|
||||
"""Resolve a user's auto-inject settings, falling back to the defaults.
|
||||
|
||||
Returns {"enabled": bool, "threshold": float, "top_k": int}, clamped to
|
||||
sane ranges (threshold to [0,1]; top_k to [1, _AUTOINJECT_MAX_TOP_K]).
|
||||
Returns {"enabled": bool, "threshold": float, "top_k": int}.
|
||||
|
||||
THE TWO NUMBERS COME FROM THE REGISTRY NOW (#4102). They used to be read and
|
||||
clamped here, and identically again in `get_writepath_config`, and again in
|
||||
three rule arms, and once more in `reply_preferences` — the duplication the
|
||||
shape ledger counts as canon #2860 across 295 of 372 siblings. That was
|
||||
tolerable while the values were shipped constants. It stops being tolerable
|
||||
once a tool is expected to MOVE them, because a tuning surface cannot be
|
||||
consistent across arms that each spell their configuration differently.
|
||||
|
||||
`enabled` stays here: it is this surface's own switch, not a tunable number,
|
||||
and the registry deliberately holds only the pair a floor-tuner touches.
|
||||
"""
|
||||
enabled_raw = await get_setting(
|
||||
user_id, AUTOINJECT_ENABLED_KEY,
|
||||
@@ -601,21 +628,11 @@ async def get_autoinject_config(user_id: int) -> dict:
|
||||
)
|
||||
enabled = enabled_raw.strip().lower() in ("true", "1", "yes", "on")
|
||||
|
||||
try:
|
||||
threshold = float(await get_setting(
|
||||
user_id, AUTOINJECT_THRESHOLD_KEY, str(AUTOINJECT_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
threshold = AUTOINJECT_DEFAULT_THRESHOLD
|
||||
threshold = min(1.0, max(0.0, threshold))
|
||||
|
||||
try:
|
||||
top_k = int(float(await get_setting(
|
||||
user_id, AUTOINJECT_TOP_K_KEY, str(AUTOINJECT_DEFAULT_TOP_K))))
|
||||
except (TypeError, ValueError):
|
||||
top_k = AUTOINJECT_DEFAULT_TOP_K
|
||||
top_k = min(_AUTOINJECT_MAX_TOP_K, max(1, top_k))
|
||||
|
||||
return {"enabled": enabled, "threshold": threshold, "top_k": top_k}
|
||||
return {
|
||||
"enabled": enabled,
|
||||
"threshold": await floor_for(user_id, "auto_inject"),
|
||||
"top_k": await budget_for(user_id, "auto_inject"),
|
||||
}
|
||||
|
||||
|
||||
def _record_kind(note) -> str:
|
||||
@@ -1024,13 +1041,8 @@ async def build_prompt_rule_hint(
|
||||
return out
|
||||
|
||||
try:
|
||||
try:
|
||||
threshold = float(await get_setting(
|
||||
user_id, PROMPTRULE_THRESHOLD_KEY,
|
||||
str(PROMPTRULE_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
threshold = PROMPTRULE_DEFAULT_THRESHOLD
|
||||
threshold = min(1.0, max(0.0, threshold))
|
||||
threshold = await floor_for(user_id, "prompt_rule")
|
||||
limit = await budget_for(user_id, "prompt_rule")
|
||||
|
||||
t0 = time.perf_counter()
|
||||
_rep: dict = {}
|
||||
@@ -1041,7 +1053,7 @@ async def build_prompt_rule_hint(
|
||||
# sessions — this surface speaks unasked, and a whole-rulebook answer
|
||||
# is only right for someone who asked the whole rulebook.
|
||||
hits = await semantic_search_rules(
|
||||
user_id, q, limit=PROMPTRULE_LIMIT, threshold=threshold,
|
||||
user_id, q, limit=limit, threshold=threshold,
|
||||
report=_rep, project_id=project_id or None,
|
||||
)
|
||||
duration_ms = (time.perf_counter() - t0) * 1000.0
|
||||
@@ -1057,7 +1069,7 @@ async def build_prompt_rule_hint(
|
||||
# and unverified for this corpus, so the zero rows are the point.
|
||||
record_retrieval(
|
||||
user_id=user_id, source="prompt_rule", query=q,
|
||||
threshold=threshold, limit=PROMPTRULE_LIMIT,
|
||||
threshold=threshold, limit=limit,
|
||||
project_id=project_id,
|
||||
is_task=None, results=fresh, duration_ms=duration_ms,
|
||||
best_available=_rep.get("best_available_score"),
|
||||
@@ -1294,52 +1306,37 @@ def concept_query(code: str) -> str:
|
||||
|
||||
|
||||
async def get_writepath_config(user_id: int) -> dict:
|
||||
"""Write-path trigger settings: its own `enabled` and `threshold`, auto-inject's top_k.
|
||||
"""Write-path trigger settings — and the two rule arms' numbers alongside.
|
||||
|
||||
The threshold OVERRIDES the inherited auto-inject value — code embeddings
|
||||
have a much higher similarity floor than prose, so the two surfaces need
|
||||
different bars. See WRITEPATH_DEFAULT_THRESHOLD for the measurements (#2223).
|
||||
top_k is still shared: "how many titles at once" means the same thing on
|
||||
both surfaces, and nothing suggests they want different ceilings.
|
||||
Three surfaces' worth of configuration arrives in one call because one hook
|
||||
request drives all three arms. They are still three SURFACES with three
|
||||
independent pairs, resolved from the registry (#4102).
|
||||
|
||||
The floors were split apart one at a time, each on its own measurement, and
|
||||
those measurements are recorded where the defaults are: WRITEPATH (#2223 —
|
||||
code embeddings sit on a much higher similarity floor than prose), RULEHINT
|
||||
(a third corpus again), TOOLRULE (#3853 — a shell command is a different
|
||||
query shape from a code payload and scores lower for the same relevance).
|
||||
|
||||
THE BUDGET IS NOW SPLIT TOO. `top_k` used to be auto-inject's outright, on
|
||||
the argument that "how many titles at once" means the same thing on both
|
||||
surfaces. It does not: this arm fires before every Write and Edit while
|
||||
auto-inject fires once a turn, so the same number buys wildly different
|
||||
amounts of attention. `write_path` inherits auto-inject's value when it has
|
||||
none of its own, so no install that tuned the shared knob loses it.
|
||||
"""
|
||||
cfg = await get_autoinject_config(user_id)
|
||||
enabled_raw = await get_setting(
|
||||
user_id, WRITEPATH_ENABLED_KEY,
|
||||
"true" if WRITEPATH_DEFAULT_ENABLED else "false",
|
||||
)
|
||||
|
||||
try:
|
||||
threshold = float(await get_setting(
|
||||
user_id, WRITEPATH_THRESHOLD_KEY, str(WRITEPATH_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
threshold = WRITEPATH_DEFAULT_THRESHOLD
|
||||
threshold = min(1.0, max(0.0, threshold))
|
||||
|
||||
try:
|
||||
rule_threshold = float(await get_setting(
|
||||
user_id, RULEHINT_THRESHOLD_KEY, str(RULEHINT_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
rule_threshold = RULEHINT_DEFAULT_THRESHOLD
|
||||
rule_threshold = min(1.0, max(0.0, rule_threshold))
|
||||
|
||||
try:
|
||||
tool_rule_threshold = float(await get_setting(
|
||||
user_id, TOOLRULE_THRESHOLD_KEY, str(TOOLRULE_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
tool_rule_threshold = TOOLRULE_DEFAULT_THRESHOLD
|
||||
tool_rule_threshold = min(1.0, max(0.0, tool_rule_threshold))
|
||||
|
||||
return {
|
||||
**cfg,
|
||||
"enabled": enabled_raw.strip().lower() in ("true", "1", "yes", "on"),
|
||||
"threshold": threshold,
|
||||
# Its own bar, for a third corpus — see RULEHINT_DEFAULT_THRESHOLD.
|
||||
"rule_threshold": rule_threshold,
|
||||
# And the COMMAND arm's own bar again, for the same reason one level
|
||||
# down: a shell command is a different query shape from a code payload
|
||||
# and scores lower for the same relevance (#3853). Separate keys, so an
|
||||
# install can move one without the other — which is the whole finding.
|
||||
"tool_rule_threshold": tool_rule_threshold,
|
||||
"threshold": await floor_for(user_id, "write_path"),
|
||||
"top_k": await budget_for(user_id, "write_path"),
|
||||
"rule_threshold": await floor_for(user_id, "write_path_rule"),
|
||||
"rule_top_k": await budget_for(user_id, "write_path_rule"),
|
||||
"tool_rule_threshold": await floor_for(user_id, "pre_tool_rule"),
|
||||
"tool_rule_top_k": await budget_for(user_id, "pre_tool_rule"),
|
||||
}
|
||||
|
||||
def _rule_band(hits: list) -> list:
|
||||
@@ -1988,7 +1985,7 @@ async def build_write_path_hint(
|
||||
rule_t0 = time.perf_counter()
|
||||
_rep_wpr: dict = {}
|
||||
hits = await semantic_search_rules(
|
||||
user_id, code or path, limit=RULEHINT_LIMIT,
|
||||
user_id, code or path, limit=cfg["rule_top_k"],
|
||||
threshold=cfg["rule_threshold"],
|
||||
report=_rep_wpr, project_id=project_id or None,
|
||||
)
|
||||
@@ -2049,7 +2046,7 @@ async def build_write_path_hint(
|
||||
# the same readout.
|
||||
record_retrieval(
|
||||
user_id=user_id, source="write_path_rule", query=code or path,
|
||||
threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
|
||||
threshold=cfg["rule_threshold"], limit=cfg["rule_top_k"],
|
||||
project_id=project_id,
|
||||
is_task=None, results=fresh, duration_ms=rule_ms,
|
||||
best_available=_rep_wpr.get("best_available_score"),
|
||||
@@ -2145,7 +2142,7 @@ async def build_tool_rule_hint(
|
||||
t0 = time.perf_counter()
|
||||
_rep_ptr: dict = {}
|
||||
hits = await semantic_search_rules(
|
||||
user_id, query, limit=RULEHINT_LIMIT,
|
||||
user_id, query, limit=cfg["tool_rule_top_k"],
|
||||
threshold=cfg["tool_rule_threshold"],
|
||||
report=_rep_ptr, project_id=project_id or None,
|
||||
)
|
||||
@@ -2168,7 +2165,7 @@ async def build_tool_rule_hint(
|
||||
# failure the arm was built to stop.
|
||||
record_retrieval(
|
||||
user_id=user_id, source="pre_tool_rule", query=query,
|
||||
threshold=cfg["tool_rule_threshold"], limit=RULEHINT_LIMIT,
|
||||
threshold=cfg["tool_rule_threshold"], limit=cfg["tool_rule_top_k"],
|
||||
project_id=project_id,
|
||||
is_task=None, results=fresh, duration_ms=duration_ms,
|
||||
best_available=_rep_ptr.get("best_available_score"),
|
||||
|
||||
@@ -61,13 +61,9 @@ import logging
|
||||
import time
|
||||
|
||||
from scribe.services.embeddings import semantic_search_rules
|
||||
from scribe.services.plugin_context import (
|
||||
REPORTPREF_DEFAULT_THRESHOLD,
|
||||
REPORTPREF_THRESHOLD_KEY,
|
||||
)
|
||||
from scribe.services.retrieval_surfaces import SURFACES, budget_for, floor_for
|
||||
from scribe.services.retrieval_telemetry import record_retrieval
|
||||
from scribe.services.rule_usage import record_rule_surfaced
|
||||
from scribe.services.settings import get_setting
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -85,16 +81,20 @@ COMPLETION_QUERY = (
|
||||
# A handful, not a menu. More than a few shape preferences for ONE kind of
|
||||
# reply would contradict each other before they helped; the limit is here to
|
||||
# keep one noisy corpus from turning a status change into a wall of text.
|
||||
LIMIT = 3
|
||||
# The STARTING budget, not the budget (#4102). Both numbers this arm runs on
|
||||
# now come from the surface registry, so the model that reads this arm's
|
||||
# telemetry can move either — which matters more here than anywhere else,
|
||||
# because a fixed query makes this arm's score a constant and a floor a hair
|
||||
# above it produces a dead arm no amount of traffic will ever reveal.
|
||||
LIMIT = SURFACES["report_preference"].budget_default
|
||||
|
||||
|
||||
async def _threshold(user_id: int) -> float:
|
||||
try:
|
||||
value = float(await get_setting(
|
||||
user_id, REPORTPREF_THRESHOLD_KEY, str(REPORTPREF_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
value = REPORTPREF_DEFAULT_THRESHOLD
|
||||
return min(1.0, max(0.0, value))
|
||||
return await floor_for(user_id, SOURCE)
|
||||
|
||||
|
||||
async def _limit(user_id: int) -> int:
|
||||
return await budget_for(user_id, SOURCE)
|
||||
|
||||
|
||||
async def completion_preferences(user_id: int, *, project_id: int | None = None) -> list[dict]:
|
||||
@@ -115,16 +115,17 @@ async def completion_preferences(user_id: int, *, project_id: int | None = None)
|
||||
"""
|
||||
try:
|
||||
threshold = await _threshold(user_id)
|
||||
limit = await _limit(user_id)
|
||||
report: dict = {}
|
||||
t0 = time.perf_counter()
|
||||
hits = await semantic_search_rules(
|
||||
user_id, COMPLETION_QUERY, limit=LIMIT, threshold=threshold,
|
||||
user_id, COMPLETION_QUERY, limit=limit, threshold=threshold,
|
||||
kind="preference", report=report, project_id=project_id,
|
||||
)
|
||||
hits = [(score, rule) for score, rule in hits if rule.kind == "preference"]
|
||||
record_retrieval(
|
||||
user_id=user_id, source=SOURCE, query=COMPLETION_QUERY,
|
||||
threshold=threshold, limit=LIMIT, project_id=project_id,
|
||||
threshold=threshold, limit=limit, project_id=project_id,
|
||||
is_task=None, results=hits,
|
||||
best_available=report.get("best_available_score"),
|
||||
best_available_id=report.get("best_available_id"),
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
"""One registry of the retrieval surfaces, and the two numbers each one has (#4102).
|
||||
|
||||
WHY THIS EXISTS
|
||||
|
||||
Six push arms each carried their own loose copy of the same shape: a settings
|
||||
key, a default, and a limit that was usually a module constant nobody could
|
||||
change. `_threshold()` was duplicated so widely that the shape ledger counts it
|
||||
as canon across 295 of 372 judged siblings (snippet #2860). That was survivable
|
||||
while the numbers were shipped constants an operator occasionally edited.
|
||||
|
||||
It stops being survivable once the numbers are meant to MOVE. The operator's
|
||||
decision for this step:
|
||||
|
||||
"the floor should be chosen and adjusted by the model using it. we've come
|
||||
back to something either fails or has to be looked at by the user we need a
|
||||
model consistent surface for the adjustment of these floor values. the user
|
||||
should be able to touch it but the model should be the thing handling it 9
|
||||
times out of 10."
|
||||
|
||||
A tuning surface cannot be "model consistent" if every arm spells its own
|
||||
configuration differently. So the arms stop owning their numbers and read them
|
||||
from here instead, and the tuning tool, the routes and the Settings UI all
|
||||
enumerate THIS table rather than hard-coding six special cases.
|
||||
|
||||
WHAT A FLOOR IS NOW, AND WHAT IT IS NOT
|
||||
|
||||
It is not a relevance judgement. Relevance is decided by the reader, which is
|
||||
the only participant that can read a trigger against a situation — that is the
|
||||
milestone's whole argument, and the injected line has always said so out loud
|
||||
("read it before deciding it does not apply").
|
||||
|
||||
A floor answers the cheaper question: *is this worth ranking at all*. `k` is
|
||||
what binds, and `k` is a BUDGET — how much of this surface's attention a
|
||||
candidate list may spend. An arm that fires before every Bash call cannot
|
||||
afford what an arm that fires once a turn can.
|
||||
|
||||
WHY THE DEFAULTS BELOW ARE STARTING POINTS AND NOT ANSWERS
|
||||
|
||||
A cosine score is a distance in BAAI/bge-small-en-v1.5's vector space, measured
|
||||
against THIS corpus. It cannot transfer to an install with different records,
|
||||
and rule 115 forbids defending a shipped default from this instance's
|
||||
telemetry — which is what made the old design unbuildable: every number was a
|
||||
guess everywhere except here, and there was no mechanism that could ever
|
||||
improve it.
|
||||
|
||||
The mechanism is the fix. Scribe ships a starting point and the means to
|
||||
correct it, so the values below carry the measurement that motivated them (see
|
||||
the long comments in `plugin_context.py`, which are kept where they are because
|
||||
they record how each number was first arrived at) without claiming to be right
|
||||
for anybody else.
|
||||
|
||||
HOW A FLOOR SHOULD ACTUALLY BE MOVED
|
||||
|
||||
By reading the records the floor refused — `retrieval_telemetry(
|
||||
near_miss_samples=N)` returns them by id — and never by the percentile alone.
|
||||
That is not a style preference; it is the one case where the two disagreed and
|
||||
was checked. `report_preference` logged 69 consecutive declines with the
|
||||
refused score 0.0006 under the bar, and every percentile said "lower it".
|
||||
Reading the refused record showed it was rule 77 "Extract intent from loose
|
||||
phrasing", a false positive, so lowering the bar would have delivered that rule
|
||||
on every completion report ever written. The statistic and the correct action
|
||||
pointed in opposite directions, and only opening the record could tell.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from scribe.services.settings import get_setting
|
||||
|
||||
# A budget nobody should be able to set past. Not a tuning value — a guard on
|
||||
# the worst case, so a mistyped setting cannot turn a menu into a wall of text.
|
||||
# Shared by every surface because it bounds the same thing everywhere: how many
|
||||
# lines a single unsolicited injection may occupy.
|
||||
MAX_BUDGET = 10
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Surface:
|
||||
"""One push arm's tunable pair, plus enough prose to tune it responsibly.
|
||||
|
||||
`asks` / `over` / `fires` are not documentation for this file — they are
|
||||
rendered by the tuning tool and the Settings UI. A floor cannot be moved
|
||||
sensibly by anyone, model or human, who does not know what the query is, what
|
||||
corpus it runs against, or how often it costs something. Those three facts
|
||||
are exactly what separates these arms from each other, and they were
|
||||
previously recoverable only by reading `plugin_context.py`.
|
||||
"""
|
||||
|
||||
name: str
|
||||
"""The telemetry `source` value, and the join key.
|
||||
|
||||
MUST equal the string this arm passes to `record_retrieval`. Everything
|
||||
useful about tuning depends on that identity: the tool that moves a floor
|
||||
and the table that says what the floor did have to be talking about the same
|
||||
arm. A test asserts it rather than a comment asking nicely.
|
||||
"""
|
||||
|
||||
floor_key: str
|
||||
floor_default: float
|
||||
budget_key: str
|
||||
budget_default: int
|
||||
asks: str
|
||||
over: str
|
||||
fires: str
|
||||
budget_falls_back_to: str = ""
|
||||
"""A budget key to inherit when this surface has none of its own set.
|
||||
|
||||
Only `write_path` uses it, and only because it USED to share auto-inject's
|
||||
`top_k` outright. Giving it a key without this would silently reset the
|
||||
budget of every install that had tuned the shared one — a behaviour change
|
||||
delivered as a default, which is the shape of regression nobody reports
|
||||
because nothing looks broken.
|
||||
"""
|
||||
|
||||
|
||||
SURFACES: dict[str, Surface] = {
|
||||
"auto_inject": Surface(
|
||||
name="auto_inject",
|
||||
floor_key="kb_autoinject_threshold",
|
||||
floor_default=0.55,
|
||||
budget_key="kb_autoinject_top_k",
|
||||
budget_default=3,
|
||||
asks="the operator's message, as they typed it",
|
||||
over="notes, snippets, processes and issues",
|
||||
fires="once per operator turn",
|
||||
),
|
||||
"write_path": Surface(
|
||||
name="write_path",
|
||||
floor_key="kb_writepath_threshold",
|
||||
floor_default=0.68,
|
||||
budget_key="kb_writepath_top_k",
|
||||
budget_default=3,
|
||||
budget_falls_back_to="kb_autoinject_top_k",
|
||||
asks="the code being written, rewritten as a concept query",
|
||||
over="snippets and recorded issues",
|
||||
fires="before every Write and Edit",
|
||||
),
|
||||
"write_path_rule": Surface(
|
||||
name="write_path_rule",
|
||||
floor_key="kb_rulehint_threshold",
|
||||
floor_default=0.72,
|
||||
budget_key="kb_rulehint_top_k",
|
||||
budget_default=5,
|
||||
asks="the code being written, against rule triggers",
|
||||
over="global rules plus the bound project's own",
|
||||
fires="before every Write and Edit",
|
||||
),
|
||||
"pre_tool_rule": Surface(
|
||||
name="pre_tool_rule",
|
||||
floor_key="kb_toolrule_threshold",
|
||||
floor_default=0.68,
|
||||
budget_key="kb_toolrule_top_k",
|
||||
budget_default=5,
|
||||
asks="the command about to run, against rule triggers",
|
||||
over="global rules plus the bound project's own",
|
||||
fires="before every Bash call — the busiest arm there is",
|
||||
),
|
||||
"prompt_rule": Surface(
|
||||
name="prompt_rule",
|
||||
floor_key="kb_promptrule_threshold",
|
||||
floor_default=0.72,
|
||||
budget_key="kb_promptrule_top_k",
|
||||
budget_default=3,
|
||||
asks="the operator's message, against rule triggers",
|
||||
over="global rules plus the bound project's own",
|
||||
fires="once per operator turn",
|
||||
),
|
||||
"report_preference": Surface(
|
||||
name="report_preference",
|
||||
floor_key="kb_reportpref_threshold",
|
||||
floor_default=0.72,
|
||||
budget_key="kb_reportpref_top_k",
|
||||
budget_default=3,
|
||||
# THE ONE FIXED QUERY, and the reason this arm behaves unlike the rest.
|
||||
# The others score something that varies per call; this one scores a
|
||||
# constant string, so its top score for a given corpus is also a
|
||||
# constant. A floor a hair above that constant is not a quiet arm, it is
|
||||
# a dead one, and no amount of traffic will ever reveal it — which is
|
||||
# precisely how this arm spent 69 calls declining the same record.
|
||||
asks="a fixed question about how to lay out a completion report",
|
||||
over="preferences",
|
||||
fires="when a task finishes",
|
||||
),
|
||||
}
|
||||
|
||||
# Reserved slots are deliberately absent. `preference_slot` and `reuse_slot`
|
||||
# borrow their parent arm's floor and are hard-limited to one hit each, because
|
||||
# their entire purpose is to guarantee a single line to a kind of record that
|
||||
# keeps losing a general score contest (#2246, #3894). A budget of "1" is the
|
||||
# feature; exposing it as tunable would invite setting it to 0 and silently
|
||||
# removing the guarantee.
|
||||
|
||||
|
||||
def surface_names() -> list[str]:
|
||||
"""Every tunable surface, in a stable order for menus and listings."""
|
||||
return list(SURFACES)
|
||||
|
||||
|
||||
def get_surface(name: str) -> Surface:
|
||||
"""Look one up, refusing an unknown name loudly.
|
||||
|
||||
A typo'd surface must not be writable. Settings keys are free-form strings
|
||||
in a generic table, so a tuning call naming `pretool_rule` would otherwise
|
||||
write a key nothing ever reads — a change that appears to succeed, reports a
|
||||
new value, and alters nothing.
|
||||
"""
|
||||
try:
|
||||
return SURFACES[name]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
f"unknown retrieval surface {name!r}. Tunable surfaces are: "
|
||||
+ ", ".join(surface_names())
|
||||
) from None
|
||||
|
||||
|
||||
async def floor_for(user_id: int, name: str) -> float:
|
||||
"""This install's current floor for a surface, clamped to [0, 1]."""
|
||||
s = get_surface(name)
|
||||
try:
|
||||
value = float(await get_setting(user_id, s.floor_key, str(s.floor_default)))
|
||||
except (TypeError, ValueError):
|
||||
value = s.floor_default
|
||||
return min(1.0, max(0.0, value))
|
||||
|
||||
|
||||
async def budget_for(user_id: int, name: str) -> int:
|
||||
"""This install's current budget for a surface, clamped to [1, MAX_BUDGET].
|
||||
|
||||
The lower clamp is 1, never 0: a surface turned off is turned off by its
|
||||
`enabled` switch, which says so. A budget of zero would be an arm that runs
|
||||
a search, logs a retrieval, and renders nothing — indistinguishable in the
|
||||
telemetry from a bar nothing cleared, which is the exact confusion this
|
||||
milestone exists to remove.
|
||||
"""
|
||||
s = get_surface(name)
|
||||
raw = await get_setting(user_id, s.budget_key, "")
|
||||
if not raw and s.budget_falls_back_to:
|
||||
raw = await get_setting(user_id, s.budget_falls_back_to, "")
|
||||
try:
|
||||
value = int(float(raw)) if raw else s.budget_default
|
||||
except (TypeError, ValueError):
|
||||
value = s.budget_default
|
||||
return min(MAX_BUDGET, max(1, value))
|
||||
Reference in New Issue
Block a user