bvandeusenandClaude Opus 5
aee24c9c1c
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m4s
CI & Build / Build & push image (push) Skipped
Milestone 416 step 6. A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor on the install keeps applying while describing nothing — and nothing anywhere says so, because the scores simply come out different and the bar goes on cutting. `CHUNKER_VERSION` already solved this for documents: stamped per row, so the backfill re-embeds precisely what is stale. The same idea, applied to the numbers: - `calibration_stamp()` — embedding model + document shape, one definition. TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half moved, because they call for different responses. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (migration 0104), stamped on every write. Nullable and NOT backfilled — "unstamped" is the honest answer for a row written before this existed, and it reports as `stale: null`, never as fine. - `current_settings` reports calibration per dial: tuned rows from their event, untouched dials from the registry default's own stamp. - `retrieval_surfaces` and the Settings panel show the mismatch. The panel renders ONLY when something is stale, so seeing it at all is the signal. - `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to be inherited by the next model so that they don't have to recalibrate a lot": the raw cosine cannot cross models, but the PERCENTILE it represented can. Measure what fraction of a surface's logged calls the old floor admitted, re-score those queries under the current model, take the value admitting the same fraction. Dry run by default; applying writes an ordinary tuning event with the arithmetic in its reason. Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be, and #4102 measured the one case where the statistic and the correct action pointed opposite ways. The load-bearing test is an ABSENCE: no chat-model identifier may appear anywhere in the calibration path. Claude produces none of these scores, so a Claude upgrade must trigger nothing — a false alarm here teaches the operator to ignore the real one on the day bge-small becomes bge-base. Backup v17 carries both columns, unfilled on the way out and on the way back: a round trip must not turn "we don't know" into a stated fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
52 lines
1.9 KiB
Python
52 lines
1.9 KiB
Python
"""retrieval_tuning_events carries the space each number was measured in (#4104)
|
|
|
|
Revision ID: 0104
|
|
Revises: 0103
|
|
Create Date: 2026-09-17
|
|
|
|
Milestone 416 step 6. A retrieval floor is a cosine distance in ONE embedding
|
|
model's geometry, computed over documents cut one particular way. Change the
|
|
model and every score moves at once; change the chunker and the same record
|
|
embeds different text. Either way a number chosen before the change is a
|
|
measurement of something that no longer exists — and today nothing records
|
|
which world it was chosen in, so the staleness is unknowable rather than
|
|
merely unknown.
|
|
|
|
`CHUNKER_VERSION` already solved exactly this for documents: stored per row, so
|
|
the startup backfill re-embeds precisely what is stale instead of wiping the
|
|
table. These two columns are that idea applied to the tuned numbers.
|
|
|
|
TWO COLUMNS, NOT ONE (rule 149). A reader has to be able to say WHICH half
|
|
moved: a new embedding model and a re-cut document shape invalidate the same
|
|
numbers for different reasons, and a fused `"<model>@<n>"` could only report
|
|
that something changed.
|
|
|
|
NULLABLE, and not backfilled. The rows already in this table were written
|
|
under something, but naming it would be inventing a fact — the honest value is
|
|
"unstamped", which is a different answer from a model name that might be wrong.
|
|
`current_settings` reports an unstamped dial as exactly that.
|
|
"""
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision = "0104"
|
|
down_revision = "0103"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column(
|
|
"retrieval_tuning_events",
|
|
sa.Column("embedding_model", sa.Text(), nullable=True),
|
|
)
|
|
op.add_column(
|
|
"retrieval_tuning_events",
|
|
sa.Column("shape_version", sa.Integer(), nullable=True),
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_column("retrieval_tuning_events", "shape_version")
|
|
op.drop_column("retrieval_tuning_events", "embedding_model")
|