feat(retrieval): a tuned number carries the space it was measured in (#4104)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped

Milestone 416 step 6. A retrieval floor is a cosine similarity, which only
means something inside one embedding model's geometry over documents cut one
particular way. Change either and every floor on the install keeps applying
while describing nothing — and nothing anywhere says so, because the scores
simply come out different and the bar goes on cutting.

`CHUNKER_VERSION` already solved this for documents: stamped per row, so the
backfill re-embeds precisely what is stale. The same idea, applied to the
numbers:

- `calibration_stamp()` — embedding model + document shape, one definition.
  TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half
  moved, because they call for different responses.
- `retrieval_tuning_events` gains `embedding_model` / `shape_version`
  (migration 0104), stamped on every write. Nullable and NOT backfilled —
  "unstamped" is the honest answer for a row written before this existed, and
  it reports as `stale: null`, never as fine.
- `current_settings` reports calibration per dial: tuned rows from their event,
  untouched dials from the registry default's own stamp.
- `retrieval_surfaces` and the Settings panel show the mismatch. The panel
  renders ONLY when something is stale, so seeing it at all is the signal.
- `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to
  be inherited by the next model so that they don't have to recalibrate a lot":
  the raw cosine cannot cross models, but the PERCENTILE it represented can.
  Measure what fraction of a surface's logged calls the old floor admitted,
  re-score those queries under the current model, take the value admitting the
  same fraction. Dry run by default; applying writes an ordinary tuning event
  with the arithmetic in its reason.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement;
it does not say what the number should be, and #4102 measured the one case
where the statistic and the correct action pointed opposite ways.

The load-bearing test is an ABSENCE: no chat-model identifier may appear
anywhere in the calibration path. Claude produces none of these scores, so a
Claude upgrade must trigger nothing — a false alarm here teaches the operator
to ignore the real one on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and on the way back:
a round trip must not turn "we don't know" into a stated fact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-17 12:45:48 -04:00
co-authored by Claude Opus 5
parent dcf800ed65
commit aee24c9c1c
13 changed files with 1002 additions and 2 deletions
+32
View File
@@ -217,6 +217,38 @@ def embedding_text(title: str | None, body: str | None) -> str:
# wipe migrations 0067/0077 had to do.
CHUNKER_VERSION = 1
# The public name of the space every score lives in, and the two facts that
# can invalidate a tuned number (#4104).
#
# EMBEDDING_MODEL is `_MODEL_NAME` under a name other modules may read. It was
# private until this step, which is precisely why nothing outside this file
# could state what space a threshold was measured in — a floor is a distance in
# THIS model's geometry and means nothing in another's.
#
# The pair is what a stamp is made of, and the pairing is the point: a score
# changes when the model changes (different geometry) OR when the document
# shape changes (different text embedded for the same record). Either one
# invalidates a number that was measured before it.
EMBEDDING_MODEL = _MODEL_NAME
def calibration_stamp() -> dict:
"""What a tuned retrieval number was measured against.
ONE definition, because the alternative is each reader assembling the pair
and one of them forgetting a half. Returned as a dict rather than a string
so a mismatch can say WHICH half moved — "the model changed" and "the
chunker changed" call for different responses, and a fused string can only
report that something did.
Deliberately says nothing about the CHAT model. A Claude upgrade changes no
score here and must never raise a recalibration prompt: a false alarm on
this surface teaches an operator to ignore the true one. `tests/
test_calibration_stamp.py` asserts that absence rather than trusting it.
"""
return {"embedding_model": EMBEDDING_MODEL, "shape_version": CHUNKER_VERSION}
# Character budget approximating the model window. Tokens-per-char varies by
# content — ~4 chars/token for prose, closer to 3 for code and tables — so 1400
# chars sits at roughly 350-470 tokens, leaving headroom for the title prefixed