feat(retrieval): a tuned number carries the space it was measured in (#4104)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
Milestone 416 step 6. A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor on the install keeps applying while describing nothing — and nothing anywhere says so, because the scores simply come out different and the bar goes on cutting. `CHUNKER_VERSION` already solved this for documents: stamped per row, so the backfill re-embeds precisely what is stale. The same idea, applied to the numbers: - `calibration_stamp()` — embedding model + document shape, one definition. TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half moved, because they call for different responses. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (migration 0104), stamped on every write. Nullable and NOT backfilled — "unstamped" is the honest answer for a row written before this existed, and it reports as `stale: null`, never as fine. - `current_settings` reports calibration per dial: tuned rows from their event, untouched dials from the registry default's own stamp. - `retrieval_surfaces` and the Settings panel show the mismatch. The panel renders ONLY when something is stale, so seeing it at all is the signal. - `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to be inherited by the next model so that they don't have to recalibrate a lot": the raw cosine cannot cross models, but the PERCENTILE it represented can. Measure what fraction of a surface's logged calls the old floor admitted, re-score those queries under the current model, take the value admitting the same fraction. Dry run by default; applying writes an ordinary tuning event with the arithmetic in its reason. Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be, and #4102 measured the one case where the statistic and the correct action pointed opposite ways. The load-bearing test is an ABSENCE: no chat-model identifier may appear anywhere in the calibration path. Claude produces none of these scores, so a Claude upgrade must trigger nothing — a false alarm here teaches the operator to ignore the real one on the day bge-small becomes bge-base. Backup v17 carries both columns, unfilled on the way out and on the way back: a round trip must not turn "we don't know" into a stated fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -178,6 +178,64 @@ interface TuningEvent {
|
||||
reason: string;
|
||||
}
|
||||
|
||||
// What a dial's number was measured against, and whether that space has moved
|
||||
// (#4104). Mirrors the `calibration` block `current_settings` returns per dial.
|
||||
// `stale` is nullable on purpose: a dial moved before Scribe recorded stamps is
|
||||
// UNKNOWN rather than fine, and rendering the two the same would hide the case
|
||||
// most likely to be wrong.
|
||||
interface DialCalibration {
|
||||
source: string;
|
||||
embedding_model: string | null;
|
||||
shape_version: number | null;
|
||||
model_changed: boolean | null;
|
||||
shape_changed: boolean | null;
|
||||
stale: boolean | null;
|
||||
}
|
||||
|
||||
interface SurfaceRow {
|
||||
surface: string;
|
||||
calibration: Record<string, DialCalibration>;
|
||||
}
|
||||
|
||||
const surfaceRows = ref<SurfaceRow[]>([]);
|
||||
|
||||
// Flattened to one line per dial that needs looking at, because the operator's
|
||||
// question is "which numbers do I no longer trust", not "tell me about each of
|
||||
// six surfaces". Anything calibrated against the live model contributes
|
||||
// nothing — the panel below renders only when this is non-empty, so a healthy
|
||||
// install never sees it. A warning that is always on the screen is one nobody
|
||||
// reads when it finally means something.
|
||||
const staleDials = computed(() =>
|
||||
surfaceRows.value.flatMap((row) =>
|
||||
Object.entries(row.calibration ?? {})
|
||||
.filter(([, cal]) => cal.stale !== false)
|
||||
.map(([dial, cal]) => ({
|
||||
key: `${row.surface}.${dial}`,
|
||||
surface: row.surface,
|
||||
dial,
|
||||
why:
|
||||
cal.source === "unstamped"
|
||||
? "changed before Scribe recorded what it was measured against"
|
||||
: cal.model_changed && cal.shape_changed
|
||||
? `measured under ${cal.embedding_model} at document shape ${cal.shape_version} — both have changed since`
|
||||
: cal.model_changed
|
||||
? `measured under ${cal.embedding_model}, which is no longer the embedding model`
|
||||
: `measured at document shape ${cal.shape_version}, and documents are cut differently now`,
|
||||
})),
|
||||
),
|
||||
);
|
||||
|
||||
async function loadSurfaces() {
|
||||
try {
|
||||
const res = await apiGet<{ surfaces: SurfaceRow[] }>("/api/retrieval/surfaces");
|
||||
surfaceRows.value = res.surfaces ?? [];
|
||||
} catch {
|
||||
// Same reasoning as the history below: unreadable calibration is not worth
|
||||
// a toast on page load, and the dials above still work.
|
||||
surfaceRows.value = [];
|
||||
}
|
||||
}
|
||||
|
||||
async function loadTuningHistory() {
|
||||
loadingTuning.value = true;
|
||||
try {
|
||||
@@ -292,6 +350,7 @@ async function saveKbInject() {
|
||||
// otherwise the panel shows a trail that is stale by exactly the change
|
||||
// the operator is looking at it to confirm.
|
||||
await loadTuningHistory();
|
||||
await loadSurfaces();
|
||||
} catch {
|
||||
toastStore.show('Failed to save auto-inject settings', 'error');
|
||||
} finally {
|
||||
@@ -767,6 +826,7 @@ onMounted(async () => {
|
||||
kbReportPrefTopK.value = allSettings.kb_reportpref_top_k;
|
||||
}
|
||||
await loadTuningHistory();
|
||||
await loadSurfaces();
|
||||
if (allSettings.kb_duplicate_threshold_snippet !== undefined) {
|
||||
kbDupThresholdSnippet.value = allSettings.kb_duplicate_threshold_snippet;
|
||||
}
|
||||
@@ -1772,6 +1832,32 @@ async function deleteUser(userId: number) {
|
||||
moves the dial with the argument attached. This panel is the other
|
||||
half of that bargain — a change made on your behalf is one you can
|
||||
read, disagree with, and set back by hand above. -->
|
||||
<!-- STALENESS (#4104). A bar is a cosine similarity, which only means
|
||||
something inside one embedding model's geometry over documents cut
|
||||
one particular way. Change either and every number above keeps
|
||||
applying while describing nothing. Shown ONLY when something has
|
||||
actually moved, so that seeing it at all is the signal. -->
|
||||
<div v-if="staleDials.length" class="calibration-warning">
|
||||
<h4 class="calibration-warning-title">
|
||||
{{ staleDials.length }}
|
||||
{{ staleDials.length === 1 ? 'value was' : 'values were' }}
|
||||
measured against something that has changed
|
||||
</h4>
|
||||
<p class="field-hint">
|
||||
These bars are similarity scores, and a similarity score only means
|
||||
something inside the model that produced it. They still apply — they
|
||||
just no longer measure what they were set to measure. Nothing is
|
||||
adjusted automatically; ask Claude to migrate them and it will carry
|
||||
each one across at the same selectivity, then check the result.
|
||||
</p>
|
||||
<ul class="calibration-list">
|
||||
<li v-for="d in staleDials" :key="d.key">
|
||||
<span class="tuning-surface">{{ d.surface }}</span>
|
||||
<span class="tuning-dial">{{ d.dial }}</span>
|
||||
<span class="calibration-why">{{ d.why }}</span>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="tuning-history">
|
||||
<h4 class="tuning-history-title">What has been tuned</h4>
|
||||
<p class="field-hint">
|
||||
@@ -4115,4 +4201,36 @@ async function deleteUser(userId: number) {
|
||||
font-size: 0.85rem;
|
||||
color: var(--fs-text-secondary);
|
||||
}
|
||||
/* Stale calibration (#4104). Tinted rather than loud: the numbers still work,
|
||||
they have merely stopped being measurements — that is a "come back to this",
|
||||
not an outage. It only renders when something is actually stale, which is
|
||||
what earns it the tint at all. */
|
||||
.calibration-warning {
|
||||
margin-top: var(--fs-space-5);
|
||||
padding: var(--fs-space-3);
|
||||
border: 1px solid var(--fs-warning);
|
||||
border-radius: var(--fs-radius-md);
|
||||
background: color-mix(in srgb, var(--fs-warning) 12%, var(--fs-surface-raised));
|
||||
}
|
||||
.calibration-warning-title {
|
||||
margin: 0 0 var(--fs-space-2);
|
||||
font-size: 0.95rem;
|
||||
color: var(--fs-warning-fg);
|
||||
}
|
||||
.calibration-list {
|
||||
list-style: none;
|
||||
margin: var(--fs-space-2) 0 0;
|
||||
padding: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--fs-space-1);
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
.calibration-list li {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
flex-wrap: wrap;
|
||||
gap: var(--fs-space-2);
|
||||
}
|
||||
.calibration-why { color: var(--fs-text-secondary); }
|
||||
</style>
|
||||
|
||||
Reference in New Issue
Block a user