feat(retrieval): a tuned number carries the space it was measured in (#4104)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
Milestone 416 step 6. A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor on the install keeps applying while describing nothing — and nothing anywhere says so, because the scores simply come out different and the bar goes on cutting. `CHUNKER_VERSION` already solved this for documents: stamped per row, so the backfill re-embeds precisely what is stale. The same idea, applied to the numbers: - `calibration_stamp()` — embedding model + document shape, one definition. TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half moved, because they call for different responses. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (migration 0104), stamped on every write. Nullable and NOT backfilled — "unstamped" is the honest answer for a row written before this existed, and it reports as `stale: null`, never as fine. - `current_settings` reports calibration per dial: tuned rows from their event, untouched dials from the registry default's own stamp. - `retrieval_surfaces` and the Settings panel show the mismatch. The panel renders ONLY when something is stale, so seeing it at all is the signal. - `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to be inherited by the next model so that they don't have to recalibrate a lot": the raw cosine cannot cross models, but the PERCENTILE it represented can. Measure what fraction of a surface's logged calls the old floor admitted, re-score those queries under the current model, take the value admitting the same fraction. Dry run by default; applying writes an ordinary tuning event with the arithmetic in its reason. Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be, and #4102 measured the one case where the statistic and the correct action pointed opposite ways. The load-bearing test is an ABSENCE: no chat-model identifier may appear anywhere in the calibration path. Claude produces none of these scores, so a Claude upgrade must trigger nothing — a false alarm here teaches the operator to ignore the real one on the day bge-small becomes bge-base. Backup v17 carries both columns, unfilled on the way out and on the way back: a round trip must not turn "we don't know" into a stated fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -103,6 +103,31 @@ class Surface:
|
||||
asks: str
|
||||
over: str
|
||||
fires: str
|
||||
measured_model: str = "BAAI/bge-small-en-v1.5"
|
||||
measured_shape: int = 1
|
||||
"""What the SHIPPED defaults above were measured against (#4104).
|
||||
|
||||
A floor is a distance in one embedding model's geometry, over documents cut
|
||||
one particular way. Either can change, and when one does every number in
|
||||
this table describes something that no longer exists.
|
||||
|
||||
TWO FIELDS, NEVER ONE FUSED STRING (rule 149). A mismatch has to be able to
|
||||
say WHICH half moved: a new embedding model and a re-cut document shape
|
||||
invalidate the same numbers for different reasons and call for different
|
||||
responses. `"<model>@<n>"` could only report that something changed, which
|
||||
is the answer nobody can act on. Same reason `calibration_stamp()` returns
|
||||
a dict and the event table gives each half its own column.
|
||||
|
||||
Recorded per surface rather than once for the module because they need not
|
||||
move together: a surface retuned after a model change carries the new stamp
|
||||
while its untouched siblings still carry the old one, and telling those
|
||||
apart is the whole job.
|
||||
|
||||
LITERALS, deliberately, rather than an import of the live values — a stamp
|
||||
says what was true when the number was chosen, so one that tracked the
|
||||
current model would always agree with it and could never report staleness.
|
||||
"""
|
||||
|
||||
budget_falls_back_to: str = ""
|
||||
"""A budget key to inherit when this surface has none of its own set.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user