feat(retrieval): a tuned number carries the space it was measured in (#4104)

Milestone 416 step 6. A retrieval floor is a cosine similarity, which only
means something inside one embedding model's geometry over documents cut one
particular way. Change either and every floor on the install keeps applying
while describing nothing — and nothing anywhere says so, because the scores
simply come out different and the bar goes on cutting.

`CHUNKER_VERSION` already solved this for documents: stamped per row, so the
backfill re-embeds precisely what is stale. The same idea, applied to the
numbers:

- `calibration_stamp()` — embedding model + document shape, one definition.
  TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half
  moved, because they call for different responses.
- `retrieval_tuning_events` gains `embedding_model` / `shape_version`
  (migration 0104), stamped on every write. Nullable and NOT backfilled —
  "unstamped" is the honest answer for a row written before this existed, and
  it reports as `stale: null`, never as fine.
- `current_settings` reports calibration per dial: tuned rows from their event,
  untouched dials from the registry default's own stamp.
- `retrieval_surfaces` and the Settings panel show the mismatch. The panel
  renders ONLY when something is stale, so seeing it at all is the signal.
- `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to
  be inherited by the next model so that they don't have to recalibrate a lot":
  the raw cosine cannot cross models, but the PERCENTILE it represented can.
  Measure what fraction of a surface's logged calls the old floor admitted,
  re-score those queries under the current model, take the value admitting the
  same fraction. Dry run by default; applying writes an ordinary tuning event
  with the arithmetic in its reason.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement;
it does not say what the number should be, and #4102 measured the one case
where the statistic and the correct action pointed opposite ways.

The load-bearing test is an ABSENCE: no chat-model identifier may appear
anywhere in the calibration path. Claude produces none of these scores, so a
Claude upgrade must trigger nothing — a false alarm here teaches the operator
to ignore the real one on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and on the way back:
a round trip must not turn "we don't know" into a stated fact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/backup.py

@@ -74,8 +74,14 @@ logger = logging.getLogger(__name__)
 # argument for them silently dropped — and from this step on those dials are
 # moved by the model, which is exactly the case where the operator needs the
 # argument to review.
+# v17 (2026-09) added retrieval_tuning_events.embedding_model / shape_version
+# (milestone 416 step 6): a floor is a distance in ONE embedding model's
+# geometry over documents cut one particular way, so the number alone cannot
+# say whether it still measures anything. Both travel NULLABLE and unfilled —
+# a row written before the stamp existed restores unstamped, because inventing
+# the model it was measured under would turn "unknown" into a stated fact.
 # Bump when the serialized schema changes.
-BACKUP_VERSION = 16
+BACKUP_VERSION = 17
 
 # Every table this backup carries, by its REAL name. Paired with _NOT_INCLUDED
 # below, these two lists must together account for the entire schema — which is
@@ -358,6 +364,12 @@ def _retrieval_tuning_event_rows(rows) -> list[dict]:
             "user_id": r.user_id, "surface": r.surface, "dial": r.dial,
             "old_value": r.old_value, "new_value": r.new_value,
             "actor": r.actor, "reason": r.reason,
+            # Carried, and NOT defaulted to the current model on the way out
+            # (#4104): a row that was unstamped when it was written is still
+            # unstamped after a round trip, and a backup that quietly filled
+            # the gap would turn "we don't know" into a stated fact.
+            "embedding_model": r.embedding_model,
+            "shape_version": r.shape_version,
             "created_at": r.created_at.isoformat() if r.created_at else None,
         }
         for r in rows
@@ -1247,6 +1259,12 @@ async def _restore_v2(data: dict) -> dict:
                 new_value=t_data.get("new_value"),
                 actor=t_data.get("actor") or "model",
                 reason=t_data.get("reason", ""),
+                # .get with no default, deliberately (v17): an archive written
+                # before the stamp existed has no key here, and None is the
+                # right answer for it — the same "unstamped" a pre-#4104 row
+                # carries in place.
+                embedding_model=t_data.get("embedding_model"),
+                shape_version=t_data.get("shape_version"),
                 created_at=_dt(t_data.get("created_at")),
             ))
             stats["retrieval_tuning_events"] += 1

src/scribe/services/embeddings.py

@@ -217,6 +217,38 @@ def embedding_text(title: str | None, body: str | None) -> str:
 # wipe migrations 0067/0077 had to do.
 CHUNKER_VERSION = 1
 
+
+# The public name of the space every score lives in, and the two facts that
+# can invalidate a tuned number (#4104).
+#
+# EMBEDDING_MODEL is `_MODEL_NAME` under a name other modules may read. It was
+# private until this step, which is precisely why nothing outside this file
+# could state what space a threshold was measured in — a floor is a distance in
+# THIS model's geometry and means nothing in another's.
+#
+# The pair is what a stamp is made of, and the pairing is the point: a score
+# changes when the model changes (different geometry) OR when the document
+# shape changes (different text embedded for the same record). Either one
+# invalidates a number that was measured before it.
+EMBEDDING_MODEL = _MODEL_NAME
+
+
+def calibration_stamp() -> dict:
+    """What a tuned retrieval number was measured against.
+
+    ONE definition, because the alternative is each reader assembling the pair
+    and one of them forgetting a half. Returned as a dict rather than a string
+    so a mismatch can say WHICH half moved — "the model changed" and "the
+    chunker changed" call for different responses, and a fused string can only
+    report that something did.
+
+    Deliberately says nothing about the CHAT model. A Claude upgrade changes no
+    score here and must never raise a recalibration prompt: a false alarm on
+    this surface teaches an operator to ignore the true one. `tests/
+    test_calibration_stamp.py` asserts that absence rather than trusting it.
+    """
+    return {"embedding_model": EMBEDDING_MODEL, "shape_version": CHUNKER_VERSION}
+
 # Character budget approximating the model window. Tokens-per-char varies by
 # content — ~4 chars/token for prose, closer to 3 for code and tables — so 1400
 # chars sits at roughly 350-470 tokens, leaving headroom for the title prefixed

src/scribe/services/retrieval_migration.py

@@ -0,0 +1,248 @@
+"""Carrying a tuned floor across an embedding-model change (#4104).
+
+THE PROBLEM
+
+A floor is a cosine similarity, and a cosine similarity is a distance in one
+model's geometry. `bge-small-en-v1.5` → `bge-base-en-v1.5` moves every score on
+the install at once, in no direction anyone can predict per record. The numbers
+in `settings` survive the swap unchanged and silently stop describing anything:
+the bar keeps applying, the telemetry keeps filling, and nothing anywhere says
+the arm is now cutting in a different place.
+
+Re-deriving six floors by hand is the alternative, and it is the thing the
+operator asked for a way out of — *"a path for thresholds to be inherited by
+the next version or different model so that they don't have to recalibrate a
+lot."*
+
+WHAT TRANSFERS, AND WHY IT IS NOT THE NUMBER
+
+The raw cosine does not transfer. **The percentile it represented does.** A
+floor's real content is a decision about SELECTIVITY — "admit roughly the top
+fifth of what this arm sees" — and that decision is about the operator's
+tolerance for noise on that surface, not about the embedder. It was true before
+the model changed and is still true after.
+
+So: measure what fraction of this surface's calls the old floor admitted, using
+the scores the old model actually produced; re-score the same queries under the
+new one; and take the value that admits the same fraction. Same decision, new
+units.
+
+WHY `best_available_score` IS THE FIGURE ON BOTH SIDES
+
+Because it is the one number that exists whether or not a call returned
+anything — the highest score the corpus offered, before the bar was applied
+(#3670). It is what `retrieval_logs` recorded under the old model, and it is
+what a re-score reproduces under the new one, so the two sides are the same
+measurement rather than two things that resemble each other.
+
+It also makes the re-score cheap to get right: `best_available_score` is
+computed over the whole candidate set BEFORE `limit` and before any
+`exclude_ids` the arm passed, so neither has to be reproduced here. Only the
+corpus FILTERS matter, which is why `_RESCORERS` below carries those and
+nothing else.
+
+WHAT THIS DOES NOT DO
+
+It does not fire by itself, and `migrate_floor` will not write anything unless
+asked twice — `apply=True` on top of having read the dry run. A model change is
+exactly the moment when every number is uncertain at once, which is the worst
+possible moment to let a statistic move six dials unattended. The percentile is
+a starting point on the new scale, in the same sense the shipped defaults are a
+starting point: better than a stale number, not a substitute for reading what
+the surface actually turned away.
+"""
+from __future__ import annotations
+
+import logging
+
+from sqlalchemy import select
+
+from scribe.models import async_session
+from scribe.models.retrieval_log import RetrievalLog
+from scribe.services.embeddings import (
+    calibration_stamp,
+    semantic_search_notes,
+    semantic_search_rules,
+)
+from scribe.services.retrieval_surfaces import floor_for, get_surface
+from scribe.services.retrieval_tuning import set_dial
+
+logger = logging.getLogger(__name__)
+
+# How many of a surface's recent calls to re-score. Every one is an embedding
+# plus a full scan, so this is a real cost — but a percentile off twenty calls
+# is noise, and the arms that matter here log hundreds a week.
+DEFAULT_SAMPLE = 200
+
+
+async def _rescore_rules(user_id: int, query: str, project_id: int | None,
+                         kind: str | None) -> float | None:
+    rep: dict = {}
+    await semantic_search_rules(
+        user_id, query, limit=1, threshold=0.0, kind=kind,
+        report=rep, project_id=project_id or None,
+    )
+    return rep.get("best_available_score")
+
+
+async def _rescore_notes(user_id: int, query: str, project_id: int | None,
+                         note_type, task_kind) -> float | None:
+    rep: dict = {}
+    await semantic_search_notes(
+        user_id, query, limit=1, threshold=0.0,
+        project_id=project_id or None, note_type=note_type,
+        task_kind=task_kind, scope="browse", report=rep,
+    )
+    return rep.get("best_available_score")
+
+
+# ONE re-scorer per surface, carrying that arm's corpus filters and nothing
+# else. These filters are stated a second time here — the arms in
+# `plugin_context` are where they are first declared — and that duplication is
+# deliberate rather than overlooked: the alternative is calling the arms
+# themselves, which build a menu, write telemetry and record records as
+# surfaced. A migration that logged two hundred fake retrievals would corrupt
+# the very table the next tuning decision reads.
+#
+# `tests/test_retrieval_migration.py` asserts every registry surface has an
+# entry, so a seventh arm cannot quietly become un-migratable.
+_RESCORERS = {
+    "auto_inject": lambda u, q, p: _rescore_notes(u, q, p, None, None),
+    "write_path": lambda u, q, p: _rescore_notes(
+        u, q, p, ("snippet", "note"), "issue"
+    ),
+    "write_path_rule": lambda u, q, p: _rescore_rules(u, q, p, None),
+    "pre_tool_rule": lambda u, q, p: _rescore_rules(u, q, p, None),
+    "prompt_rule": lambda u, q, p: _rescore_rules(u, q, p, None),
+    "report_preference": lambda u, q, p: _rescore_rules(u, q, p, "preference"),
+}
+
+
+def _floor_admitting(scores: list[float], fraction: float) -> float:
+    """The floor that admits `fraction` of `scores`, on this scale.
+
+    Deliberately exact rather than interpolated: with the scores sorted
+    highest-first, the k-th one IS the bar that admits exactly k. An
+    interpolated quantile would return a number no observed call sits on, which
+    is harder to sanity-check against the sample it came from.
+
+    A fraction rounding to zero returns a floor just above the best score seen —
+    an arm that admitted nothing keeps admitting nothing, rather than being
+    quietly reopened by a migration.
+    """
+    ranked = sorted(scores, reverse=True)
+    k = int(round(fraction * len(ranked)))
+    if k <= 0:
+        return min(1.0, ranked[0] + 1e-6)
+    return ranked[min(k, len(ranked)) - 1]
+
+
+async def migrate_floor(
+    user_id: int,
+    surface: str,
+    *,
+    sample: int = DEFAULT_SAMPLE,
+    apply: bool = False,
+) -> dict:
+    """Recompute one surface's floor on the current model, preserving selectivity.
+
+    Returns the working: how many calls were sampled, what fraction the old
+    floor admitted, and what value admits the same fraction now. Writes nothing
+    unless `apply=True`, and when it does it writes an ordinary tuning event
+    with the arithmetic in its reason — a migrated floor is reviewable and
+    revertible on exactly the same terms as one a reader chose.
+    """
+    get_surface(surface)                          # refuses an unknown name
+    rescore = _RESCORERS.get(surface)
+    if rescore is None:
+        raise ValueError(
+            f"no re-scorer for surface {surface!r}. A surface that cannot be "
+            "re-scored cannot be migrated — add it to _RESCORERS beside the "
+            "arm's own corpus filters."
+        )
+
+    old_floor = await floor_for(user_id, surface)
+    async with async_session() as session:
+        rows = (await session.execute(
+            select(
+                RetrievalLog.query,
+                RetrievalLog.project_id,
+                RetrievalLog.best_available_score,
+            )
+            .where(
+                RetrievalLog.source == surface,
+                RetrievalLog.user_id == user_id,
+                RetrievalLog.best_available_score.is_not(None),
+                RetrievalLog.query.is_not(None),
+                RetrievalLog.query != "",
+            )
+            .order_by(RetrievalLog.id.desc())
+            .limit(max(1, int(sample)))
+        )).all()
+
+    if not rows:
+        # Not an error. A surface with no logged calls has no evidence of what
+        # its floor was doing, and inventing a migration for it would be the
+        # exact failure this module's docstring warns about.
+        return {
+            "surface": surface, "migrated": False,
+            "why": "no logged calls carry a best_available_score for this "
+                   "surface, so there is no old distribution to preserve",
+            "sampled": 0, "old_floor": old_floor,
+        }
+
+    old_scores = [float(r.best_available_score) for r in rows]
+    admitted = sum(1 for s in old_scores if s >= old_floor)
+    fraction = admitted / len(old_scores)
+
+    new_scores: list[float] = []
+    for r in rows:
+        score = await rescore(user_id, r.query, r.project_id)
+        if score is not None:
+            new_scores.append(float(score))
+
+    if not new_scores:
+        # The corpus answered nothing for any sampled query. Almost always an
+        # embedder that has not finished backfilling under the new model —
+        # migrating from it would set every floor off an empty distribution.
+        return {
+            "surface": surface, "migrated": False,
+            "why": "re-scoring returned nothing for any sampled query — the "
+                   "corpus is probably not embedded under the current model yet",
+            "sampled": len(rows), "old_floor": old_floor,
+            "old_admit_rate": round(fraction, 4),
+        }
+
+    proposed = round(min(1.0, max(0.0, _floor_admitting(new_scores, fraction))), 4)
+    stamp = calibration_stamp()
+    reason = (
+        f"Migrated across a calibration change, preserving selectivity: the old "
+        f"floor {old_floor} admitted {admitted} of {len(old_scores)} sampled "
+        f"calls ({fraction:.1%}); {proposed} admits the same share of "
+        f"{len(new_scores)} queries re-scored under "
+        f"{stamp['embedding_model']}/shape {stamp['shape_version']}. The "
+        f"percentile is what carried across, not the number — spot-check the "
+        f"arm before trusting it."
+    )
+
+    result = {
+        "surface": surface,
+        "migrated": False,
+        "sampled": len(rows),
+        "rescored": len(new_scores),
+        "old_floor": old_floor,
+        "old_admit_rate": round(fraction, 4),
+        "old_score_range": [round(min(old_scores), 4), round(max(old_scores), 4)],
+        "new_score_range": [round(min(new_scores), 4), round(max(new_scores), 4)],
+        "proposed_floor": proposed,
+        "calibration": stamp,
+        "reason": reason,
+    }
+    if not apply:
+        return result
+
+    result["migrated"] = True
+    result["applied"] = await set_dial(
+        user_id, surface, "floor", proposed, reason=reason, actor="model",
+    )
+    return result

src/scribe/services/retrieval_surfaces.py

@@ -103,6 +103,31 @@ class Surface:
     asks: str
     over: str
     fires: str
+    measured_model: str = "BAAI/bge-small-en-v1.5"
+    measured_shape: int = 1
+    """What the SHIPPED defaults above were measured against (#4104).
+
+    A floor is a distance in one embedding model's geometry, over documents cut
+    one particular way. Either can change, and when one does every number in
+    this table describes something that no longer exists.
+
+    TWO FIELDS, NEVER ONE FUSED STRING (rule 149). A mismatch has to be able to
+    say WHICH half moved: a new embedding model and a re-cut document shape
+    invalidate the same numbers for different reasons and call for different
+    responses. `"<model>@<n>"` could only report that something changed, which
+    is the answer nobody can act on. Same reason `calibration_stamp()` returns
+    a dict and the event table gives each half its own column.
+
+    Recorded per surface rather than once for the module because they need not
+    move together: a surface retuned after a model change carries the new stamp
+    while its untouched siblings still carry the old one, and telling those
+    apart is the whole job.
+
+    LITERALS, deliberately, rather than an import of the live values — a stamp
+    says what was true when the number was chosen, so one that tracked the
+    current model would always agree with it and could never report staleness.
+    """
+
     budget_falls_back_to: str = ""
     """A budget key to inherit when this surface has none of its own set.
 

src/scribe/services/retrieval_tuning.py

@@ -47,6 +47,7 @@ from sqlalchemy import select
 
 from scribe.models import async_session
 from scribe.models.retrieval_tuning import RetrievalTuningEvent
+from scribe.services.embeddings import calibration_stamp
 from scribe.services.retrieval_surfaces import (
     MAX_BUDGET,
     budget_for,
@@ -86,6 +87,50 @@ def _clean_reason(reason: str) -> str:
     return text
 
 
+def _calibration(row, s, live: dict) -> dict:
+    """What space one dial's number was chosen in, and whether that space moved.
+
+    Three sources, and they are not interchangeable:
+
+    - `tuned`     — the dial was moved after #4104 and the event carries its
+                    stamp. The only case where the answer is known.
+    - `shipped`   — the dial has never been moved, so the number in force is
+                    the registry default, and the registry records what THAT
+                    was measured against (`Surface.measured_model`).
+    - `unstamped` — the dial was moved before this step existed. It was
+                    measured under something; naming it would be inventing a
+                    fact, so `stale` is None rather than True or False.
+                    "Unknown" and "fine" must not render the same.
+
+    `model_changed` and `shape_changed` are reported apart (rule 149) because
+    they call for different responses: a new embedding model means every number
+    is a distance in a geometry that no longer exists, while a re-cut document
+    shape means the same records now embed different text. A caller that only
+    ever sees `stale: true` cannot tell those apart.
+    """
+    if row is None:
+        model, shape, source = s.measured_model, s.measured_shape, "shipped"
+    elif row.embedding_model is None and row.shape_version is None:
+        return {
+            "source": "unstamped", "embedding_model": None,
+            "shape_version": None, "model_changed": None,
+            "shape_changed": None, "stale": None,
+        }
+    else:
+        model, shape, source = row.embedding_model, row.shape_version, "tuned"
+
+    model_changed = model != live["embedding_model"]
+    shape_changed = shape != live["shape_version"]
+    return {
+        "source": source,
+        "embedding_model": model,
+        "shape_version": shape,
+        "model_changed": model_changed,
+        "shape_changed": shape_changed,
+        "stale": model_changed or shape_changed,
+    }
+
+
 async def current_settings(user_id: int) -> list[dict]:
     """Every tunable surface with its live pair and its last stated reason.
 
@@ -94,7 +139,16 @@ async def current_settings(user_id: int) -> list[dict]:
     often — because a floor cannot be moved sensibly without those three — and
     the reason last given, so the next change argues with the last one instead
     of overwriting it blind.
+
+    From #4104 it also carries `calibration` per dial: the embedding model and
+    document shape the number in force was measured in, and whether either has
+    moved since. NOTHING AUTO-RETUNES on the strength of it. A stale stamp says
+    a number is a measurement of a space that no longer exists — it does not
+    say what the number should be now, and the one time a statistic was allowed
+    to answer that question it was wrong (see the module docstring). The stamp
+    is here so a reader knows which floors to go and re-measure.
     """
+    live = calibration_stamp()
     out: list[dict] = []
     async with async_session() as session:
         for name in surface_names():
@@ -126,6 +180,13 @@ async def current_settings(user_id: int) -> list[dict]:
                 "last_change": {
                     dial: last[dial].to_dict() for dial in DIALS if dial in last
                 },
+                # Always present for BOTH dials, unlike `last_change`: an
+                # untouched dial still has a number in force, and that number
+                # was still measured in some space. Reported, never acted on —
+                # see the note below on why nothing auto-retunes.
+                "calibration": {
+                    dial: _calibration(last.get(dial), s, live) for dial in DIALS
+                },
             })
     return out
 
@@ -179,9 +240,16 @@ async def set_dial(
 
     await set_setting(user_id, key, stored)
     async with async_session() as session:
+        # Stamped with the space this number was chosen in (#4104). Read at
+        # write time rather than passed in: the caller measuring a floor and
+        # the caller recording it are the same call, so there is no window in
+        # which they could disagree.
+        stamp = calibration_stamp()
         session.add(RetrievalTuningEvent(
             user_id=user_id, surface=surface, dial=dial,
             old_value=old, new_value=applied, actor=actor, reason=text,
+            embedding_model=stamp["embedding_model"],
+            shape_version=stamp["shape_version"],
         ))
         await session.commit()