fix(tests): two readers moved, and the settings guard now checks the registry (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 43s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m2s
CI & Build / Build & push image (push) Skipped

CI on 09b4845 caught both, and the second is an improvement rather than a
repair.

`test_services_reply_preferences` patched `reply_preferences.get_setting`,
which the registry refactor removed — its floor and budget are resolved through
`retrieval_surfaces` now. Its key-independence test also asserted the arm asks
for exactly one key; it asks for two, because both of its numbers are its own
since this step, so the assertion names both and adds a check that the registry
and the module constant still agree about the floor key. They writing different
keys is the failure where the Settings form saves one string and the arm reads
another.

`test_settings_defaults_agree` parsed `plugin_context.py` for a bare
module-level float, and those constants now alias the registry. Rather than
teach the regex about aliases, the six retrieval floors are keyed on their
SURFACE NAME and read from the registry directly — which is strictly better for
this guard: a surface name is also its telemetry source, so a row names the same
arm the readout does, and a floor cannot be checked against a stale constant
that happened to keep its old value. `PLAN_MATCH_DEFAULT_THRESHOLD` is not a
push surface and keeps the older shape, with a note saying why.

Added while there: `test_every_tunable_surface_has_a_control`, derived from the
registry, so a seventh surface arrives as a failing test rather than as a number
only the model can reach (rules 25, 27).

Also lands the audit trail this step needs — `retrieval_tuning_events` (model +
migration 0103) and `services/retrieval_tuning.py`. The tool layer on top is
the next commit; the table is here because `reason` being REQUIRED is the whole
guardrail, and the schema is where that starts. A number moved silently leaves
nothing for the operator to review or disagree with, and the operator's decision
is that the model moves these "9 times out of 10".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-17 11:20:47 -04:00
co-authored by Claude Opus 5
parent 09b48457ff
commit 003bfd7a0a
6 changed files with 463 additions and 29 deletions
+208
View File
@@ -0,0 +1,208 @@
"""Moving a retrieval surface's floor or budget, with the argument attached (#4102).
WHY THIS EXISTS
The operator's decision for milestone 416 step 4:
"the floor should be chosen and adjusted by the model using it. we've come
back to something either fails or has to be looked at by the user we need a
model consistent surface for the adjustment of these floor values. the user
should be able to touch it but the model should be the thing handling it 9
times out of 10."
`retrieval_surfaces` made the six arms describe their numbers the same way.
This module is the write half: one call that moves one dial on one surface,
records what it was, what it became, who moved it and why, and refuses to do
any of that without a reason.
WHY A REASON IS REQUIRED
Because the alternative was tried and measured. The milestone originally listed
self-tuning as a non-goal on the strength of one case: `report_preference`
logged 69 consecutive declines with the refused record 0.0006 under the bar, and
every percentile in the readout said "lower it". Reading the refused record
showed it was rule 77 "Extract intent from loose phrasing" — a false positive —
so lowering the bar would have attached that rule to every completion report
ever written. The statistic and the correct action pointed in opposite
directions.
What separated them was opening the record. A required `reason` is the cheapest
mechanism that makes that step happen: a caller who must write down why has to
have looked, and a caller who writes down something wrong has left the operator
a sentence to disagree with. A number moved silently leaves nothing.
WHAT THIS DELIBERATELY DOES NOT DO
It does not decide anything itself. There is no rule here that reads a
percentile and picks a value, and that absence is the design — the non-goal that
survived is *statistical* auto-tuning, precisely because the statistic was the
thing that was wrong. The judgement stays with the reader; this module only
makes the judgement recordable and reversible.
"""
from __future__ import annotations
import logging
from sqlalchemy import select
from scribe.models import async_session
from scribe.models.retrieval_tuning import RetrievalTuningEvent
from scribe.services.retrieval_surfaces import (
MAX_BUDGET,
budget_for,
floor_for,
get_surface,
surface_names,
)
from scribe.services.settings import set_setting
logger = logging.getLogger(__name__)
DIALS = ("floor", "budget")
# Long enough to say what was read and what it showed; short enough that nobody
# pastes a telemetry dump in. The number is not a measurement — it is the point
# at which "0.66" stops being an acceptable answer to "why".
_MIN_REASON_CHARS = 20
def _clean_reason(reason: str) -> str:
"""The guardrail, enforced here rather than by the column.
`reason` is NOT NULL in the schema, which "" satisfies. A required field
that accepts an empty string is a formality, and this one is the whole
mechanism — see the module docstring.
"""
text = (reason or "").strip()
if len(text) < _MIN_REASON_CHARS:
raise ValueError(
"reason is required, and has to say what you read. A floor moved "
"without a stated basis is a number nobody can review or revert. "
"Name the evidence: which surface's telemetry, and what the refused "
"records actually were — `retrieval_telemetry(near_miss_samples=N)` "
"returns them by id, and reading them is the step that separates a "
"real miss from a bar doing its job."
)
return text
async def current_settings(user_id: int) -> list[dict]:
"""Every tunable surface with its live pair and its last stated reason.
The read side of the tuning surface, and shaped for a reader who is about to
change something: the value, what the arm asks and over what corpus and how
often — because a floor cannot be moved sensibly without those three — and
the reason last given, so the next change argues with the last one instead
of overwriting it blind.
"""
out: list[dict] = []
async with async_session() as session:
for name in surface_names():
s = get_surface(name)
rows = (
await session.execute(
select(RetrievalTuningEvent)
.where(
RetrievalTuningEvent.surface == name,
RetrievalTuningEvent.user_id == user_id,
)
.order_by(RetrievalTuningEvent.created_at.desc())
.limit(len(DIALS))
)
).scalars().all()
last = {r.dial: r for r in rows}
out.append({
"surface": name,
"floor": await floor_for(user_id, name),
"budget": await budget_for(user_id, name),
"floor_default": s.floor_default,
"budget_default": s.budget_default,
"asks": s.asks,
"over": s.over,
"fires": s.fires,
# Absent rather than empty when a surface has never been moved:
# "still on the shipped starting point" is a real state and
# should not render as a blank reason somebody wrote.
"last_change": {
dial: last[dial].to_dict() for dial in DIALS if dial in last
},
})
return out
async def set_dial(
user_id: int,
surface: str,
dial: str,
value: float,
*,
reason: str,
actor: str = "model",
) -> dict:
"""Move one dial on one surface, recording the change and its argument.
Returns the applied value alongside the previous one, so a caller can see
that a clamp bit rather than assuming the number it sent is the number in
force — the failure being avoided is a tool reporting success for a value
the registry silently corrected.
"""
s = get_surface(surface) # refuses an unknown name
if dial not in DIALS:
raise ValueError(f"dial must be one of {DIALS}, got {dial!r}")
if actor not in ("model", "human"):
raise ValueError(f"actor must be 'model' or 'human', got {actor!r}")
text = _clean_reason(reason)
if dial == "floor":
old = await floor_for(user_id, surface)
applied = min(1.0, max(0.0, float(value)))
key, stored = s.floor_key, str(applied)
else:
old = float(await budget_for(user_id, surface))
applied = float(min(MAX_BUDGET, max(1, int(float(value)))))
key, stored = s.budget_key, str(int(applied))
await set_setting(user_id, key, stored)
async with async_session() as session:
session.add(RetrievalTuningEvent(
user_id=user_id, surface=surface, dial=dial,
old_value=old, new_value=applied, actor=actor, reason=text,
))
await session.commit()
return {
"surface": surface,
"dial": dial,
"previous": old,
"applied": applied,
# True when the registry corrected what was asked for. Said out loud
# because a caller that believes it set 1.4 will read the next
# telemetry as evidence about a bar that was never in force.
"clamped": abs(applied - float(value)) > 1e-9,
"reason": text,
"actor": actor,
}
async def tuning_history(
user_id: int, *, surface: str | None = None, limit: int = 20
) -> list[dict]:
"""What has been moved, newest first — the operator's review surface.
Scoped to one surface when asked, because the question is almost always
"why is THIS arm set like this", and an unscoped list buries one surface's
two changes under another's twenty.
"""
if surface is not None:
get_surface(surface) # refuse a typo on the read too
async with async_session() as session:
q = (
select(RetrievalTuningEvent)
.where(RetrievalTuningEvent.user_id == user_id)
.order_by(RetrievalTuningEvent.created_at.desc())
.limit(max(1, min(int(limit), 200)))
)
if surface is not None:
q = q.where(RetrievalTuningEvent.surface == surface)
rows = (await session.execute(q)).scalars().all()
return [r.to_dict() for r in rows]