feat(retrieval): a tuned number carries the space it was measured in (#4104)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
Milestone 416 step 6. A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor on the install keeps applying while describing nothing — and nothing anywhere says so, because the scores simply come out different and the bar goes on cutting. `CHUNKER_VERSION` already solved this for documents: stamped per row, so the backfill re-embeds precisely what is stale. The same idea, applied to the numbers: - `calibration_stamp()` — embedding model + document shape, one definition. TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half moved, because they call for different responses. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (migration 0104), stamped on every write. Nullable and NOT backfilled — "unstamped" is the honest answer for a row written before this existed, and it reports as `stale: null`, never as fine. - `current_settings` reports calibration per dial: tuned rows from their event, untouched dials from the registry default's own stamp. - `retrieval_surfaces` and the Settings panel show the mismatch. The panel renders ONLY when something is stale, so seeing it at all is the signal. - `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to be inherited by the next model so that they don't have to recalibrate a lot": the raw cosine cannot cross models, but the PERCENTILE it represented can. Measure what fraction of a surface's logged calls the old floor admitted, re-score those queries under the current model, take the value admitting the same fraction. Dry run by default; applying writes an ordinary tuning event with the arithmetic in its reason. Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be, and #4102 measured the one case where the statistic and the correct action pointed opposite ways. The load-bearing test is an ABSENCE: no chat-model identifier may appear anywhere in the calibration path. Claude produces none of these scores, so a Claude upgrade must trigger nothing — a false alarm here teaches the operator to ignore the real one on the day bge-small becomes bge-base. Backup v17 carries both columns, unfilled on the way out and on the way back: a round trip must not turn "we don't know" into a stated fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -74,8 +74,14 @@ logger = logging.getLogger(__name__)
|
||||
# argument for them silently dropped — and from this step on those dials are
|
||||
# moved by the model, which is exactly the case where the operator needs the
|
||||
# argument to review.
|
||||
# v17 (2026-09) added retrieval_tuning_events.embedding_model / shape_version
|
||||
# (milestone 416 step 6): a floor is a distance in ONE embedding model's
|
||||
# geometry over documents cut one particular way, so the number alone cannot
|
||||
# say whether it still measures anything. Both travel NULLABLE and unfilled —
|
||||
# a row written before the stamp existed restores unstamped, because inventing
|
||||
# the model it was measured under would turn "unknown" into a stated fact.
|
||||
# Bump when the serialized schema changes.
|
||||
BACKUP_VERSION = 16
|
||||
BACKUP_VERSION = 17
|
||||
|
||||
# Every table this backup carries, by its REAL name. Paired with _NOT_INCLUDED
|
||||
# below, these two lists must together account for the entire schema — which is
|
||||
@@ -358,6 +364,12 @@ def _retrieval_tuning_event_rows(rows) -> list[dict]:
|
||||
"user_id": r.user_id, "surface": r.surface, "dial": r.dial,
|
||||
"old_value": r.old_value, "new_value": r.new_value,
|
||||
"actor": r.actor, "reason": r.reason,
|
||||
# Carried, and NOT defaulted to the current model on the way out
|
||||
# (#4104): a row that was unstamped when it was written is still
|
||||
# unstamped after a round trip, and a backup that quietly filled
|
||||
# the gap would turn "we don't know" into a stated fact.
|
||||
"embedding_model": r.embedding_model,
|
||||
"shape_version": r.shape_version,
|
||||
"created_at": r.created_at.isoformat() if r.created_at else None,
|
||||
}
|
||||
for r in rows
|
||||
@@ -1247,6 +1259,12 @@ async def _restore_v2(data: dict) -> dict:
|
||||
new_value=t_data.get("new_value"),
|
||||
actor=t_data.get("actor") or "model",
|
||||
reason=t_data.get("reason", ""),
|
||||
# .get with no default, deliberately (v17): an archive written
|
||||
# before the stamp existed has no key here, and None is the
|
||||
# right answer for it — the same "unstamped" a pre-#4104 row
|
||||
# carries in place.
|
||||
embedding_model=t_data.get("embedding_model"),
|
||||
shape_version=t_data.get("shape_version"),
|
||||
created_at=_dt(t_data.get("created_at")),
|
||||
))
|
||||
stats["retrieval_tuning_events"] += 1
|
||||
|
||||
Reference in New Issue
Block a user