feat(retrieval): a tuned number carries the space it was measured in (#4104)

Milestone 416 step 6. A retrieval floor is a cosine similarity, which only
means something inside one embedding model's geometry over documents cut one
particular way. Change either and every floor on the install keeps applying
while describing nothing — and nothing anywhere says so, because the scores
simply come out different and the bar goes on cutting.

`CHUNKER_VERSION` already solved this for documents: stamped per row, so the
backfill re-embeds precisely what is stale. The same idea, applied to the
numbers:

- `calibration_stamp()` — embedding model + document shape, one definition.
  TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half
  moved, because they call for different responses.
- `retrieval_tuning_events` gains `embedding_model` / `shape_version`
  (migration 0104), stamped on every write. Nullable and NOT backfilled —
  "unstamped" is the honest answer for a row written before this existed, and
  it reports as `stale: null`, never as fine.
- `current_settings` reports calibration per dial: tuned rows from their event,
  untouched dials from the registry default's own stamp.
- `retrieval_surfaces` and the Settings panel show the mismatch. The panel
  renders ONLY when something is stale, so seeing it at all is the signal.
- `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to
  be inherited by the next model so that they don't have to recalibrate a lot":
  the raw cosine cannot cross models, but the PERCENTILE it represented can.
  Measure what fraction of a surface's logged calls the old floor admitted,
  re-score those queries under the current model, take the value admitting the
  same fraction. Dry run by default; applying writes an ordinary tuning event
  with the arithmetic in its reason.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement;
it does not say what the number should be, and #4102 measured the one case
where the statistic and the correct action pointed opposite ways.

The load-bearing test is an ABSENCE: no chat-model identifier may appear
anywhere in the calibration path. Claude produces none of these scores, so a
Claude upgrade must trigger nothing — a false alarm here teaches the operator
to ignore the real one on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and on the way back:
a round trip must not turn "we don't know" into a stated fact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

tests/test_calibration_stamp.py

@@ -0,0 +1,191 @@
+"""A tuned number carries the space it was measured in — and only that (#4104).
+
+WHY THIS EXISTS
+
+Milestone 416 step 6 answers *"a path for thresholds to be inherited by the next
+version or different model so that they don't have to recalibrate a lot."* The
+mechanism is a stamp: every tuning event records the embedding model and
+document shape the number was chosen under, and a mismatch is reported rather
+than left to be noticed.
+
+THE ONE THAT MATTERS MOST IS THE ABSENCE
+
+`test_no_chat_model_identifier_reaches_the_calibration_path` is the load-bearing
+guard here, and it is a guard against a plausible mistake rather than a
+hypothetical one. These floors live in BAAI/bge-small-en-v1.5's vector space.
+The CHAT model — Claude — produces none of these scores and changes none of
+them, so a Claude upgrade must trigger nothing at all. If it ever did, the
+operator would be asked to recalibrate six dials for no reason, learn that this
+surface cries wolf, and ignore it on the day `bge-small` → `bge-base` actually
+moves every score on the install. A false alarm here does not cost a
+notification; it costs the real alarm.
+
+The rest pins the honest-unknown behaviour: a row written before stamps existed
+reports as `unstamped` with `stale: None`, never as fine and never as a model
+name somebody guessed at.
+"""
+import re
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from scribe.services import embeddings as emb
+from scribe.services import retrieval_tuning as rt
+from scribe.services.retrieval_surfaces import SURFACES, get_surface
+from tests.helpers import make_mock_session, tool_doc
+
+# Names for the thing that talks, as opposed to the thing that embeds. Any of
+# these appearing in the calibration path means a chat-model change could move a
+# stamp — see the module docstring for why that is the expensive failure.
+_CHAT_MODEL_WORDS = (
+    "claude", "opus", "sonnet", "haiku", "gpt", "llama", "mistral", "gemini",
+    "anthropic", "openai", "chat_model", "chatmodel", "model_version",
+)
+
+
+def test_calibration_stamp_names_the_embedder_and_the_shape():
+    stamp = emb.calibration_stamp()
+    assert stamp == {
+        "embedding_model": emb.EMBEDDING_MODEL,
+        "shape_version": emb.CHUNKER_VERSION,
+    }
+    # Two keys, never one fused string (rule 149): a mismatch has to be able to
+    # say WHICH half moved, because a new embedder and a re-cut document
+    # invalidate the same numbers for different reasons.
+    assert len(stamp) == 2
+
+
+def test_the_stamp_tracks_the_embedding_model_constant_not_a_copy_of_it():
+    """A second literal would drift, and drift here reads as a model change."""
+    with patch.object(emb, "EMBEDDING_MODEL", "some/other-model"):
+        assert emb.calibration_stamp()["embedding_model"] == "some/other-model"
+    with patch.object(emb, "CHUNKER_VERSION", 99):
+        assert emb.calibration_stamp()["shape_version"] == 99
+
+
+def test_no_chat_model_identifier_reaches_the_calibration_path():
+    """THE GUARD. A Claude upgrade must move nothing in this path.
+
+    Structural rather than behavioural on purpose: the failure is someone
+    *adding* a chat-model field in good faith ("surely the model matters"), and
+    no behavioural test catches a field that has not been written yet. Asserted
+    over the source of every module that produces or consumes a stamp.
+    """
+    import inspect
+
+    from scribe.services import retrieval_migration as rm
+
+    for label, obj in (
+        ("calibration_stamp", emb.calibration_stamp),
+        ("_calibration", rt._calibration),
+        ("migrate_floor", rm.migrate_floor),
+    ):
+        src = inspect.getsource(obj).lower()
+        # Comments and docstrings here legitimately discuss the chat model in
+        # order to rule it out, so strip prose and assert on code only.
+        code = "\n".join(
+            line.split("#", 1)[0]
+            for line in re.sub(r'""".*?"""', "", src, flags=re.S).splitlines()
+        )
+        for word in _CHAT_MODEL_WORDS:
+            assert word not in code, f"{label} names the chat model: {word!r}"
+
+
+@pytest.mark.asyncio
+async def test_a_moved_dial_is_written_with_the_live_stamp():
+    session = make_mock_session()
+    with patch.object(rt, "async_session", MagicMock(return_value=session)), \
+         patch.object(rt, "set_setting", AsyncMock()), \
+         patch.object(rt, "floor_for", AsyncMock(return_value=0.72)), \
+         patch.object(rt, "budget_for", AsyncMock(return_value=3)), \
+         patch.object(rt, "calibration_stamp",
+                      MagicMock(return_value={"embedding_model": "m/x",
+                                              "shape_version": 7})):
+        await rt.set_dial(
+            1, "prompt_rule", "floor", 0.66,
+            reason="read the five refused records; four were genuine matches",
+        )
+    event = session.add.call_args[0][0]
+    assert event.embedding_model == "m/x"
+    assert event.shape_version == 7
+
+
+def _row(model="BAAI/bge-small-en-v1.5", shape=1):
+    return MagicMock(embedding_model=model, shape_version=shape)
+
+
+def test_an_untouched_dial_reports_the_shipped_default_it_is_still_on():
+    s = get_surface("prompt_rule")
+    live = {"embedding_model": s.measured_model, "shape_version": s.measured_shape}
+    cal = rt._calibration(None, s, live)
+    assert cal["source"] == "shipped"
+    assert cal["stale"] is False
+
+
+def test_a_dial_moved_before_stamps_existed_is_unknown_not_fine():
+    """`stale: None`, because "we don't know" and "it's fine" are not the same.
+
+    Collapsing them would hide the dials MOST likely to be wrong — the ones
+    somebody tuned longest ago.
+    """
+    s = get_surface("prompt_rule")
+    cal = rt._calibration(_row(model=None, shape=None), s, emb.calibration_stamp())
+    assert cal["source"] == "unstamped"
+    assert cal["stale"] is None
+    assert cal["model_changed"] is None and cal["shape_changed"] is None
+
+
+def test_a_model_change_and_a_shape_change_are_reported_apart():
+    """Rule 149. Two conditions, two answers — they call for different work."""
+    s = get_surface("prompt_rule")
+    live = {"embedding_model": "new/model", "shape_version": 1}
+    cal = rt._calibration(_row(model="old/model", shape=1), s, live)
+    assert (cal["model_changed"], cal["shape_changed"], cal["stale"]) == (
+        True, False, True,
+    )
+
+    live = {"embedding_model": "old/model", "shape_version": 2}
+    cal = rt._calibration(_row(model="old/model", shape=1), s, live)
+    assert (cal["model_changed"], cal["shape_changed"], cal["stale"]) == (
+        False, True, True,
+    )
+
+
+def test_a_dial_tuned_under_the_live_stamp_is_not_stale():
+    s = get_surface("prompt_rule")
+    live = emb.calibration_stamp()
+    cal = rt._calibration(
+        _row(model=live["embedding_model"], shape=live["shape_version"]), s, live,
+    )
+    assert cal["source"] == "tuned"
+    assert cal["stale"] is False
+
+
+def test_every_surface_ships_a_stamp_for_its_defaults():
+    """A default with no stamp cannot be told from one measured yesterday."""
+    for name, s in SURFACES.items():
+        assert s.measured_model, f"{name} default has no measured model"
+        assert isinstance(s.measured_shape, int), f"{name} shape is not a version"
+
+
+def test_the_registry_stamp_is_a_literal_not_the_live_value():
+    """It says what WAS true, so it must not follow the current constants.
+
+    A field that tracked `EMBEDDING_MODEL` would agree with it forever and could
+    never report the one thing it exists to report.
+    """
+    with patch.object(emb, "EMBEDDING_MODEL", "some/other-model"):
+        assert get_surface("prompt_rule").measured_model != "some/other-model"
+
+
+def test_the_tool_says_nothing_auto_retunes():
+    """The contract an agent reads. A stale stamp is a prompt to MEASURE.
+
+    Without this, the obvious next move on seeing `stale: true` is to move the
+    dial — which is tuning from a statistic, the exact failure #4102 measured
+    pointing the wrong way.
+    """
+    doc = tool_doc("scribe.mcp.tools.retrieval_tuning", "retrieval_surfaces")
+    assert "calibration" in doc
+    assert "NOTHING IS RETUNED AUTOMATICALLY" in doc
+    assert "unstamped" in doc

tests/test_retrieval_migration.py

@@ -0,0 +1,153 @@
+"""Carrying a floor across a calibration change (#4104).
+
+WHAT THIS PINS
+
+  1. **The percentile is what transfers, not the number.** A floor's content is
+     a decision about selectivity; the cosine expressing it is units. So a
+     migration that admitted 30% before admits 30% after, whatever the new
+     scores look like.
+  2. **Nothing is written unless asked twice.** `apply` defaults to False. A
+     model change makes every number uncertain at once, which is the worst
+     moment to let a statistic move six dials unattended.
+  3. **Missing evidence is a refusal, not a guess.** No logged calls, or a
+     corpus that re-scores to nothing, returns `migrated: False` with a reason —
+     never a floor computed off an empty distribution, which is how an install
+     mid-backfill would end up with every bar at zero.
+  4. **Every registry surface can be migrated.** A seventh arm that nobody adds
+     a re-scorer for is one whose floor silently cannot survive a model change.
+"""
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from scribe.services import retrieval_migration as rm
+from scribe.services.retrieval_surfaces import SURFACES
+from tests.helpers import make_mock_session, tool_doc
+
+
+def _logs(pairs):
+    """Rows as `migrate_floor` reads them: (query, project_id, old score)."""
+    return [MagicMock(query=q, project_id=p, best_available_score=s)
+            for q, p, s in pairs]
+
+
+def _session_with(rows):
+    session = make_mock_session()
+    session.execute.return_value.all.return_value = rows
+    return session
+
+
+def test_every_surface_has_a_rescorer():
+    """Otherwise a surface's floor cannot cross a model change at all."""
+    assert set(rm._RESCORERS) == set(SURFACES)
+
+
+def test_the_floor_that_admits_a_fraction_is_an_observed_score():
+    scores = [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.05]
+    # 30% of ten is three; the third-best score is the bar that admits exactly
+    # those three. Exact rather than interpolated, so the answer can be checked
+    # against the sample it came from.
+    assert rm._floor_admitting(scores, 0.3) == 0.7
+    assert sum(1 for s in scores if s >= 0.7) == 3
+
+
+def test_a_surface_that_admitted_nothing_keeps_admitting_nothing():
+    """A migration must not quietly reopen an arm the operator had shut."""
+    scores = [0.5, 0.4, 0.3]
+    assert rm._floor_admitting(scores, 0.0) > max(scores)
+
+
+@pytest.mark.asyncio
+async def test_selectivity_is_preserved_across_a_scale_change():
+    """The whole idea, on numbers that move a long way.
+
+    Old scores cluster near 0.7 with a bar at 0.72 admitting two of five. New
+    scores sit far lower — a different geometry — and the proposal is the value
+    that admits two of five there, not anything resembling 0.72.
+    """
+    rows = _logs([
+        ("q1", 2, 0.80), ("q2", 2, 0.75), ("q3", 2, 0.70),
+        ("q4", 2, 0.60), ("q5", 2, 0.50),
+    ])
+    new = {"q1": 0.42, "q2": 0.38, "q3": 0.31, "q4": 0.22, "q5": 0.10}
+    rescore = AsyncMock(side_effect=lambda u, q, p: new[q])
+
+    with patch.object(rm, "async_session", MagicMock(return_value=_session_with(rows))), \
+         patch.object(rm, "floor_for", AsyncMock(return_value=0.72)), \
+         patch.dict(rm._RESCORERS, {"prompt_rule": rescore}):
+        out = await rm.migrate_floor(1, "prompt_rule")
+
+    assert out["old_admit_rate"] == 0.4          # 0.80 and 0.75 cleared 0.72
+    assert out["proposed_floor"] == 0.38         # admits 0.42 and 0.38 — also two
+    assert out["migrated"] is False              # dry run by default
+
+
+@pytest.mark.asyncio
+async def test_a_dry_run_writes_nothing():
+    rows = _logs([("q1", None, 0.9), ("q2", None, 0.1)])
+    with patch.object(rm, "async_session", MagicMock(return_value=_session_with(rows))), \
+         patch.object(rm, "floor_for", AsyncMock(return_value=0.5)), \
+         patch.object(rm, "set_dial", AsyncMock()) as set_dial, \
+         patch.dict(rm._RESCORERS, {"prompt_rule": AsyncMock(return_value=0.4)}):
+        out = await rm.migrate_floor(1, "prompt_rule")
+    set_dial.assert_not_called()
+    assert "proposed_floor" in out
+
+
+@pytest.mark.asyncio
+async def test_applying_writes_an_ordinary_tuning_event_with_the_arithmetic():
+    """A migrated floor is reviewable and revertible like any other change."""
+    rows = _logs([("q1", None, 0.9), ("q2", None, 0.1)])
+    with patch.object(rm, "async_session", MagicMock(return_value=_session_with(rows))), \
+         patch.object(rm, "floor_for", AsyncMock(return_value=0.5)), \
+         patch.object(rm, "set_dial", AsyncMock(return_value={})) as set_dial, \
+         patch.dict(rm._RESCORERS, {"prompt_rule": AsyncMock(return_value=0.4)}):
+        out = await rm.migrate_floor(1, "prompt_rule", apply=True)
+
+    assert out["migrated"] is True
+    kwargs = set_dial.call_args.kwargs
+    reason = kwargs["reason"]
+    # The reason has to carry the working, not just the verdict — it is what the
+    # operator reads to decide whether to keep the number.
+    assert "0.5" in reason and "sampled" in reason
+    assert kwargs["actor"] == "model"
+
+
+@pytest.mark.asyncio
+async def test_no_logged_calls_refuses_rather_than_inventing_a_distribution():
+    with patch.object(rm, "async_session", MagicMock(return_value=_session_with([]))), \
+         patch.object(rm, "floor_for", AsyncMock(return_value=0.5)), \
+         patch.object(rm, "set_dial", AsyncMock()) as set_dial:
+        out = await rm.migrate_floor(1, "prompt_rule", apply=True)
+    assert out["migrated"] is False
+    assert "no logged calls" in out["why"]
+    set_dial.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_a_corpus_that_rescores_to_nothing_refuses():
+    """The mid-backfill case: every bar would otherwise be set off no data."""
+    rows = _logs([("q1", None, 0.9), ("q2", None, 0.8)])
+    with patch.object(rm, "async_session", MagicMock(return_value=_session_with(rows))), \
+         patch.object(rm, "floor_for", AsyncMock(return_value=0.5)), \
+         patch.object(rm, "set_dial", AsyncMock()) as set_dial, \
+         patch.dict(rm._RESCORERS, {"prompt_rule": AsyncMock(return_value=None)}):
+        out = await rm.migrate_floor(1, "prompt_rule", apply=True)
+    assert out["migrated"] is False
+    assert "not embedded" in out["why"]
+    set_dial.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_an_unknown_surface_is_refused():
+    with pytest.raises(ValueError):
+        await rm.migrate_floor(1, "promptrule")
+
+
+def test_the_tool_says_it_is_a_starting_point_and_defaults_to_a_dry_run():
+    doc = tool_doc("scribe.mcp.tools.retrieval_tuning", "migrate_retrieval_floor")
+    assert "DRY RUN BY DEFAULT" in doc
+    # Percentile-preserving carries the old floor's wrongness forward faithfully.
+    # A reader who misses that will treat a migrated number as a measured one.
+    assert "STARTING POINT" in doc.upper()
+    assert "stale" in doc

tests/test_services_backup.py

@@ -23,7 +23,7 @@ def test_backup_version_is_current():
 
     (Named for the number it asserted until v10, which is exactly the drift a
     name-carrying-a-value invites; it now says what it checks.)"""
-    assert backup.BACKUP_VERSION == 16
+    assert backup.BACKUP_VERSION == 17
 
 
 def _exportable_note(**over):