feat(retrieval): a tuned number carries the space it was measured in (#4104)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped

Milestone 416 step 6. A retrieval floor is a cosine similarity, which only
means something inside one embedding model's geometry over documents cut one
particular way. Change either and every floor on the install keeps applying
while describing nothing — and nothing anywhere says so, because the scores
simply come out different and the bar goes on cutting.

`CHUNKER_VERSION` already solved this for documents: stamped per row, so the
backfill re-embeds precisely what is stale. The same idea, applied to the
numbers:

- `calibration_stamp()` — embedding model + document shape, one definition.
  TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half
  moved, because they call for different responses.
- `retrieval_tuning_events` gains `embedding_model` / `shape_version`
  (migration 0104), stamped on every write. Nullable and NOT backfilled —
  "unstamped" is the honest answer for a row written before this existed, and
  it reports as `stale: null`, never as fine.
- `current_settings` reports calibration per dial: tuned rows from their event,
  untouched dials from the registry default's own stamp.
- `retrieval_surfaces` and the Settings panel show the mismatch. The panel
  renders ONLY when something is stale, so seeing it at all is the signal.
- `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to
  be inherited by the next model so that they don't have to recalibrate a lot":
  the raw cosine cannot cross models, but the PERCENTILE it represented can.
  Measure what fraction of a surface's logged calls the old floor admitted,
  re-score those queries under the current model, take the value admitting the
  same fraction. Dry run by default; applying writes an ordinary tuning event
  with the arithmetic in its reason.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement;
it does not say what the number should be, and #4102 measured the one case
where the statistic and the correct action pointed opposite ways.

The load-bearing test is an ABSENCE: no chat-model identifier may appear
anywhere in the calibration path. Claude produces none of these scores, so a
Claude upgrade must trigger nothing — a false alarm here teaches the operator
to ignore the real one on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and on the way back:
a round trip must not turn "we don't know" into a stated fact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
2026-09-17 12:45:48 -04:00
co-authored by Claude Opus 5
parent dcf800ed65
commit aee24c9c1c
13 changed files with 1002 additions and 2 deletions
+118
View File
@@ -178,6 +178,64 @@ interface TuningEvent {
reason: string;
}
// What a dial's number was measured against, and whether that space has moved
// (#4104). Mirrors the `calibration` block `current_settings` returns per dial.
// `stale` is nullable on purpose: a dial moved before Scribe recorded stamps is
// UNKNOWN rather than fine, and rendering the two the same would hide the case
// most likely to be wrong.
interface DialCalibration {
source: string;
embedding_model: string | null;
shape_version: number | null;
model_changed: boolean | null;
shape_changed: boolean | null;
stale: boolean | null;
}
interface SurfaceRow {
surface: string;
calibration: Record<string, DialCalibration>;
}
const surfaceRows = ref<SurfaceRow[]>([]);
// Flattened to one line per dial that needs looking at, because the operator's
// question is "which numbers do I no longer trust", not "tell me about each of
// six surfaces". Anything calibrated against the live model contributes
// nothing — the panel below renders only when this is non-empty, so a healthy
// install never sees it. A warning that is always on the screen is one nobody
// reads when it finally means something.
const staleDials = computed(() =>
surfaceRows.value.flatMap((row) =>
Object.entries(row.calibration ?? {})
.filter(([, cal]) => cal.stale !== false)
.map(([dial, cal]) => ({
key: `${row.surface}.${dial}`,
surface: row.surface,
dial,
why:
cal.source === "unstamped"
? "changed before Scribe recorded what it was measured against"
: cal.model_changed && cal.shape_changed
? `measured under ${cal.embedding_model} at document shape ${cal.shape_version} — both have changed since`
: cal.model_changed
? `measured under ${cal.embedding_model}, which is no longer the embedding model`
: `measured at document shape ${cal.shape_version}, and documents are cut differently now`,
})),
),
);
async function loadSurfaces() {
try {
const res = await apiGet<{ surfaces: SurfaceRow[] }>("/api/retrieval/surfaces");
surfaceRows.value = res.surfaces ?? [];
} catch {
// Same reasoning as the history below: unreadable calibration is not worth
// a toast on page load, and the dials above still work.
surfaceRows.value = [];
}
}
async function loadTuningHistory() {
loadingTuning.value = true;
try {
@@ -292,6 +350,7 @@ async function saveKbInject() {
// otherwise the panel shows a trail that is stale by exactly the change
// the operator is looking at it to confirm.
await loadTuningHistory();
await loadSurfaces();
} catch {
toastStore.show('Failed to save auto-inject settings', 'error');
} finally {
@@ -767,6 +826,7 @@ onMounted(async () => {
kbReportPrefTopK.value = allSettings.kb_reportpref_top_k;
}
await loadTuningHistory();
await loadSurfaces();
if (allSettings.kb_duplicate_threshold_snippet !== undefined) {
kbDupThresholdSnippet.value = allSettings.kb_duplicate_threshold_snippet;
}
@@ -1772,6 +1832,32 @@ async function deleteUser(userId: number) {
moves the dial with the argument attached. This panel is the other
half of that bargain a change made on your behalf is one you can
read, disagree with, and set back by hand above. -->
<!-- STALENESS (#4104). A bar is a cosine similarity, which only means
something inside one embedding model's geometry over documents cut
one particular way. Change either and every number above keeps
applying while describing nothing. Shown ONLY when something has
actually moved, so that seeing it at all is the signal. -->
<div v-if="staleDials.length" class="calibration-warning">
<h4 class="calibration-warning-title">
{{ staleDials.length }}
{{ staleDials.length === 1 ? 'value was' : 'values were' }}
measured against something that has changed
</h4>
<p class="field-hint">
These bars are similarity scores, and a similarity score only means
something inside the model that produced it. They still apply — they
just no longer measure what they were set to measure. Nothing is
adjusted automatically; ask Claude to migrate them and it will carry
each one across at the same selectivity, then check the result.
</p>
<ul class="calibration-list">
<li v-for="d in staleDials" :key="d.key">
<span class="tuning-surface">{{ d.surface }}</span>
<span class="tuning-dial">{{ d.dial }}</span>
<span class="calibration-why">{{ d.why }}</span>
</li>
</ul>
</div>
<div class="tuning-history">
<h4 class="tuning-history-title">What has been tuned</h4>
<p class="field-hint">
@@ -4115,4 +4201,36 @@ async function deleteUser(userId: number) {
font-size: 0.85rem;
color: var(--fs-text-secondary);
}
/* Stale calibration (#4104). Tinted rather than loud: the numbers still work,
they have merely stopped being measurements — that is a "come back to this",
not an outage. It only renders when something is actually stale, which is
what earns it the tint at all. */
.calibration-warning {
margin-top: var(--fs-space-5);
padding: var(--fs-space-3);
border: 1px solid var(--fs-warning);
border-radius: var(--fs-radius-md);
background: color-mix(in srgb, var(--fs-warning) 12%, var(--fs-surface-raised));
}
.calibration-warning-title {
margin: 0 0 var(--fs-space-2);
font-size: 0.95rem;
color: var(--fs-warning-fg);
}
.calibration-list {
list-style: none;
margin: var(--fs-space-2) 0 0;
padding: 0;
display: flex;
flex-direction: column;
gap: var(--fs-space-1);
font-size: 0.85rem;
}
.calibration-list li {
display: flex;
align-items: baseline;
flex-wrap: wrap;
gap: var(--fs-space-2);
}
.calibration-why { color: var(--fs-text-secondary); }
</style>