fix(tests): two readers moved, and the settings guard now checks the registry (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 43s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m2s
CI & Build / Build & push image (push) Skipped

CI on 09b4845 caught both, and the second is an improvement rather than a
repair.

`test_services_reply_preferences` patched `reply_preferences.get_setting`,
which the registry refactor removed — its floor and budget are resolved through
`retrieval_surfaces` now. Its key-independence test also asserted the arm asks
for exactly one key; it asks for two, because both of its numbers are its own
since this step, so the assertion names both and adds a check that the registry
and the module constant still agree about the floor key. They writing different
keys is the failure where the Settings form saves one string and the arm reads
another.

`test_settings_defaults_agree` parsed `plugin_context.py` for a bare
module-level float, and those constants now alias the registry. Rather than
teach the regex about aliases, the six retrieval floors are keyed on their
SURFACE NAME and read from the registry directly — which is strictly better for
this guard: a surface name is also its telemetry source, so a row names the same
arm the readout does, and a floor cannot be checked against a stale constant
that happened to keep its old value. `PLAN_MATCH_DEFAULT_THRESHOLD` is not a
push surface and keeps the older shape, with a note saying why.

Added while there: `test_every_tunable_surface_has_a_control`, derived from the
registry, so a seventh surface arrives as a failing test rather than as a number
only the model can reach (rules 25, 27).

Also lands the audit trail this step needs — `retrieval_tuning_events` (model +
migration 0103) and `services/retrieval_tuning.py`. The tool layer on top is
the next commit; the table is here because `reason` being REQUIRED is the whole
guardrail, and the schema is where that starts. A number moved silently leaves
nothing for the operator to review or disagree with, and the operator's decision
is that the model moves these "9 times out of 10".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-17 11:20:47 -04:00
co-authored by Claude Opus 5
parent 09b48457ff
commit 003bfd7a0a
6 changed files with 463 additions and 29 deletions
+18 -5
View File
@@ -9,6 +9,9 @@ import re
from unittest.mock import AsyncMock, MagicMock, patch
M = "scribe.services.reply_preferences"
# Its two numbers are resolved through the registry now (#4102), so the
# settings reader to patch lives there rather than in the arm's own module.
RS = "scribe.services.retrieval_surfaces"
def _rule(rid, kind="preference"):
@@ -26,7 +29,7 @@ async def _run(hits, *, searched=True, raises=None):
search_mock = AsyncMock(side_effect=search)
with patch(f"{M}.semantic_search_rules", search_mock), \
patch(f"{M}.get_setting", AsyncMock(return_value="0.72")), \
patch(f"{RS}.get_setting", AsyncMock(return_value="0.72")), \
patch(f"{M}.record_retrieval") as logged, \
patch(f"{M}.record_rule_surfaced") as surfaced:
out = await completion_preferences(7, project_id=3)
@@ -93,7 +96,7 @@ def test_its_bar_is_its_own_key_not_the_prompt_arm_s(monkeypatch):
asked: list[str] = []
async def get_setting(user_id, key, default):
async def get_setting(user_id, key, default=""):
asked.append(key)
return default
@@ -101,16 +104,26 @@ def test_its_bar_is_its_own_key_not_the_prompt_arm_s(monkeypatch):
kw["report"].update({"searched": True})
return []
with patch(f"{M}.get_setting", AsyncMock(side_effect=get_setting)), \
with patch(f"{RS}.get_setting", AsyncMock(side_effect=get_setting)), \
patch(f"{M}.semantic_search_rules", AsyncMock(side_effect=search)), \
patch(f"{M}.record_retrieval"):
asyncio.run(completion_preferences(7))
assert asked == [plugin_context.REPORTPREF_THRESHOLD_KEY]
assert plugin_context.REPORTPREF_THRESHOLD_KEY != plugin_context.PROMPTRULE_THRESHOLD_KEY, (
# Both numbers are its own since #4102 — a floor AND a budget — so the arm
# asks for two keys, and neither may be the prompt arm's.
from scribe.services.retrieval_surfaces import get_surface
mine = get_surface("report_preference")
assert asked == [mine.floor_key, mine.budget_key]
theirs = get_surface("prompt_rule")
assert mine.floor_key != theirs.floor_key, (
"the two keys are the same string again, so the settings form has one "
"dial driving two arms — which is the defect, whatever the value is"
)
assert mine.floor_key == plugin_context.REPORTPREF_THRESHOLD_KEY, (
"the registry and the module constant disagree about this arm's key, "
"so the Settings form would write one and the arm would read the other"
)
def test_the_query_assumes_no_particular_domain():