feat(retrieval): the model moves its own floors, and says why (#4102)

Milestone 416 step 4's write half. `retrieval_surfaces.py` made the six
push arms describe their `{floor, budget}` the same way; this adds the
three MCP tools that let the model READ that and change it, and the
backup sections that carry the reasons.

The operator's decision, which this implements:

    "the floor should be chosen and adjusted by the model using it… the
    user should be able to touch it but the model should be the thing
    handling it 9 times out of 10."

WHY A REASON IS REQUIRED, AND WHY THE TOOL ARGUES AGAINST PERCENTILES

The milestone originally listed self-tuning as a non-goal on one
measured case, and that case is now the tool's docstring rather than a
prohibition: `report_preference` logged 69 consecutive declines with
the refused record 0.0006 under the bar, and every percentile said
"lower it". The refused record was rule 77 "Extract intent from loose
phrasing" matched against a query about report layout — a false
positive. Lowering would have attached that rule to every completion
report ever written.

What separated the statistic from the correct action was OPENING the
record. So `tune_retrieval` refuses a blank or perfunctory reason,
tells the caller to read `retrieval_telemetry(near_miss_samples=5)`
and the record ids it names, and carries that 69-decline example — an
abstract warning loses to a number. The non-goal that survives is
*statistical* auto-tuning; nothing here reads a percentile and picks a
value.

BACKUP (v16), which is what CI caught

`retrieval_tuning_events` was neither backed up nor excluded, and
#2293's guard said so. It is backed up: `settings` already carried the
numbers, so dropping this would restore an install with six moved
dials and no argument for any of them — precisely the state the table
exists to prevent, and worse now that the model is the one moving
them. One `_retrieval_tuning_event_rows` builder called from both
exporters (snippet #2851); `user_id` travels because a restore has to
remap it, which is why the row builder is not the model's `to_dict()`.
`surface` is a registry name rather than a foreign key, so the history
survives a restore into an install whose ids all differ.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/backup.py

@@ -13,6 +13,7 @@ from scribe.models.rule_version import RuleVersion
 from scribe.models.design_system import DesignSystem, DesignToken
 from scribe.models.note_usage import NoteUsageEvent
 from scribe.models.rule_usage import RuleUsageEvent
+from scribe.models.retrieval_tuning import RetrievalTuningEvent
 from scribe.models.canonical_system import CanonicalSystem
 from scribe.models.rulebook import RuleRelation, rule_systems as rule_systems_t
 from scribe.models.code_shape import CodeShape, CodeShapeEvent, CodeShapeUse
@@ -67,8 +68,14 @@ logger = logging.getLogger(__name__)
 # topic_suppressions with their tables (milestone 414): a rule's scope is its
 # home now. Older archives carrying those sections still restore — the keys are
 # simply not read — as do the subscribe_rulebooks inception choices they hold.
+# v16 (2026-09) added retrieval_tuning_events (milestone 416): the REASON each
+# retrieval floor and budget is where it is. `settings` already carried the
+# numbers, so leaving this behind would restore six moved dials with the
+# argument for them silently dropped — and from this step on those dials are
+# moved by the model, which is exactly the case where the operator needs the
+# argument to review.
 # Bump when the serialized schema changes.
-BACKUP_VERSION = 15
+BACKUP_VERSION = 16
 
 # Every table this backup carries, by its REAL name. Paired with _NOT_INCLUDED
 # below, these two lists must together account for the entire schema — which is
@@ -100,6 +107,12 @@ _BACKED_UP = [
     # accumulated, so a restore that dropped it would silently reset the
     # measurement to zero while everything still looked fine.
     "rule_usage_events",
+    # v16 (2026-09): why each retrieval floor and budget is where it is
+    # (milestone 416). The values live in `settings` and already travelled; the
+    # argument for them had nowhere to go. Now that the model moves these dials
+    # on the operator's behalf, a restore that kept the numbers and dropped the
+    # reasons would leave an install tuned by nobody it can name.
+    "retrieval_tuning_events",
 ]
 
 # Tables intentionally NOT in the backup, surfaced in the payload so the gap is
@@ -189,6 +202,9 @@ _COLUMN_EXCLUSIONS: dict[str, set[str]] = {
     "note_usage_events": {"id"},
     # Same as the note twin: the surrogate key is re-issued on insert.
     "rule_usage_events": {"id"},
+    # Same again — and everything else travels, because each remaining column
+    # is part of the argument: what moved, from what, to what, by whom, why.
+    "retrieval_tuning_events": {"id"},
     "design_systems": {"deleted_at", "deleted_batch_id", "created_at", "updated_at"},
     "design_tokens": {"deleted_at", "deleted_batch_id", "created_at", "updated_at"},
     "repo_bindings": {"id", "created_at", "updated_at"},
@@ -329,6 +345,25 @@ def _rule_usage_event_rows(rows) -> list[dict]:
     ]
 
 
+def _retrieval_tuning_event_rows(rows) -> list[dict]:
+    """The record of why a retrieval dial is where it is (#4102).
+
+    Not `to_dict()`: that method serves the MCP reader, which already knows
+    whose install it is asking about, so it omits `user_id`. A restore has to
+    remap it, and a row that arrived without it would have to be dropped or
+    guessed at.
+    """
+    return [
+        {
+            "user_id": r.user_id, "surface": r.surface, "dial": r.dial,
+            "old_value": r.old_value, "new_value": r.new_value,
+            "actor": r.actor, "reason": r.reason,
+            "created_at": r.created_at.isoformat() if r.created_at else None,
+        }
+        for r in rows
+    ]
+
+
 def _code_shape_rows(rows) -> list[dict]:
     return [r.to_dict() for r in rows]
 
@@ -617,6 +652,12 @@ async def export_full_backup() -> dict:
         rule_usage_events = (
             await session.execute(select(RuleUsageEvent))
         ).scalars().all()
+        # Oldest first, so a restored history reads in the order the dials
+        # actually moved — the sequence IS the argument when a surface has been
+        # walked up and down.
+        retrieval_tuning_events = (await session.execute(
+            select(RetrievalTuningEvent).order_by(RetrievalTuningEvent.id)
+        )).scalars().all()
         repo_bindings = (await session.execute(select(RepoBinding))).scalars().all()
         code_shapes = (await session.execute(select(CodeShape))).scalars().all()
         code_shape_events = (await session.execute(
@@ -661,6 +702,9 @@ async def export_full_backup() -> dict:
         "design_tokens": _design_token_rows(design_tokens),
         "note_usage_events": _usage_event_rows(usage_events),
         "rule_usage_events": _rule_usage_event_rows(rule_usage_events),
+        "retrieval_tuning_events": _retrieval_tuning_event_rows(
+            retrieval_tuning_events
+        ),
         "repo_bindings": _repo_binding_rows(repo_bindings),
         "note_supersessions": _note_supersession_rows(supersessions),
         "code_shapes": _code_shape_rows(code_shapes),
@@ -795,6 +839,15 @@ async def export_user_backup(user_id: int) -> dict:
         rule_usage_events = (await session.execute(
             select(RuleUsageEvent).where(RuleUsageEvent.rule_id.in_(_rule_ids))
         )).scalars().all() if _rule_ids else []
+        # Scoped on user_id, and here that IS the right column — unlike the
+        # rule usage events directly above. These record changes to this user's
+        # OWN retrieval settings, which is what `user_id` means on this table;
+        # there is no second owner to route around.
+        retrieval_tuning_events = (await session.execute(
+            select(RetrievalTuningEvent)
+            .where(RetrievalTuningEvent.user_id == user_id)
+            .order_by(RetrievalTuningEvent.id)
+        )).scalars().all()
         rule_relations = (await session.execute(
             select(RuleRelation).where(
                 RuleRelation.from_rule_id.in_(_rule_ids),
@@ -836,6 +889,9 @@ async def export_user_backup(user_id: int) -> dict:
         "design_tokens": _design_token_rows(design_tokens),
         "note_usage_events": _usage_event_rows(usage_events),
         "rule_usage_events": _rule_usage_event_rows(rule_usage_events),
+        "retrieval_tuning_events": _retrieval_tuning_event_rows(
+            retrieval_tuning_events
+        ),
         "repo_bindings": _repo_binding_rows(repo_bindings),
         "note_supersessions": _note_supersession_rows(supersessions),
         "code_shapes": _code_shape_rows(code_shapes),
@@ -975,6 +1031,7 @@ async def _restore_v2(data: dict) -> dict:
         "note_supersessions": 0, "code_shapes": 0, "code_shape_events": 0,
         "code_shape_uses": 0, "canonical_systems": 0,
         "rule_systems": 0, "rule_relations": 0, "rule_versions": 0,
+        "retrieval_tuning_events": 0,
     }
 
     async with async_session() as session:
@@ -1169,6 +1226,31 @@ async def _restore_v2(data: dict) -> dict:
             session.add(Setting(user_id=mapped_uid, key=s_data["key"], value=s_data.get("value", "")))
             stats["settings"] += 1
 
+        # 8b. Retrieval tuning history (v16) — restored beside the settings it
+        # explains, and for the same reason: the numbers above are the state,
+        # these rows are the argument for it. From milestone 416 those dials
+        # move on the operator's behalf, so an install restored with the values
+        # and without the reasons is one tuned by nobody it can name.
+        #
+        # No id remapping beyond the user: `surface` is a registry NAME, not a
+        # foreign key, which is what lets this history survive a restore into
+        # an install whose row ids all differ.
+        for t_data in data.get("retrieval_tuning_events", []):
+            mapped_uid = user_id_map.get(t_data.get("user_id") or 0)
+            if mapped_uid is None:
+                continue
+            session.add(RetrievalTuningEvent(
+                user_id=mapped_uid,
+                surface=t_data.get("surface", ""),
+                dial=t_data.get("dial", ""),
+                old_value=t_data.get("old_value"),
+                new_value=t_data.get("new_value"),
+                actor=t_data.get("actor") or "model",
+                reason=t_data.get("reason", ""),
+                created_at=_dt(t_data.get("created_at")),
+            ))
+            stats["retrieval_tuning_events"] += 1
+
         # 9. Rulebooks (v3)
         for rb_data in data.get("rulebooks", []):
             mapped_uid = user_id_map.get(rb_data.get("owner_user_id", 0))