6 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	42360f616c	fix(telemetry): silence is only an answer where the ledger was watching (#4268) ... `band_hugs_floor` compares an arm's weakest tenth against its floor, and #4225 made it SUSPEND rather than soften when the floor moved inside the window — because across a change the scores and the bar come from two different populations. It reads `floor_moves_since`, which answers "did this arm's floor move", and it read an empty answer as "no, it held steady". Those are the same answer only where the ledger was watching. Before an arm's first floor row there is nothing to move, nothing to report, and no way to tell a steady floor from an unrecorded one. The suspension was reading absence of evidence as evidence of absence, and the symptom is the one #4225 documented: a band "-0.0208 above its floor" — an impossible negative distance, printed with the suspension silent. Every install passes through this. The ledger's first row for an arm is written when that install first boots the release that records baselines, so any window longer than the install is old reaches back past it. The lowered-floor direction hides: the gap comes out comfortably positive and reads as a clean bill of health. `floor_history_gaps(since)` answers the question its companion cannot: which arms' floor history does not REACH the start of the window. Arms come from `surface_names()`, not from the ledger — an arm the ledger has never heard of is exactly the one at risk, so it cannot be the ledger that decides which arms get asked about. Baselines count here, which is the one place the two deliberately disagree. A baseline records a default without changing it, so it is not a move and `floor_moves_since` filters it out. It IS the ledger beginning to observe the arm, and from that moment silence genuinely means the floor held — filtering it out here would suspend the band check forever on every install that has never tuned. `floor_history_unknown` sits between the known move and the band, so an arm with a date to give gives it. Two sentences, because the remedies differ: a date says ask again with a smaller `days`; no history at all says there is nothing to wait for, and names what starts the record. Why now: #4261 measures the work-log change by reading these warnings, and every window for the next month opens before this ledger's first row. Measuring against an instrument that prints a number it cannot support is the #4225 trap one level up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 14:38:16 -04:00
bvandeusenandClaude Opus 5	512d0326a0	fix(telemetry): a floor that moved inside the window makes the band check a comparison of two populations (#4225) ... `retrieval_telemetry(days=30)` reported, for write_path_rule: "the weakest tenth of what this arm returns scores 0.6984, only -0.0216 above its floor of 0.72" A negative distance above something. The tenth percentile of what an arm RETURNED cannot sit below the floor that gates what it may return — not inside one population. MEASURED CAUSE. write_path_rule's floor was 0.68 until 2026-09-02, when 2385100 (#3318) raised the shipped default to 0.72. The window opened 2026-08-22, so six days of it are calls made under the old bar; top_score.min for the surface is exactly 0.68, the old bar still in the sample. AND THE CHANGE LEFT NO TRACE THE READOUT COULD SEE. retrieval_tuning_events records dial turns — a person or a model choosing a number. It was silent about the other way a floor moves: somebody edits floor_default and ships it. retrieval_tuning_history returned {"events": []} and retrieval_surfaces said last_change: {}, source: "shipped". All true, and all of it silent about a floor that had in fact moved. THE RAISE ANNOUNCED ITSELF. A LOWERED FLOOR WOULD NOT: the gap comes out comfortably positive and reads as a clean bill of health on a sample that half predates the bar being judged. Both directions are now pinned. So the check is SUSPENDED, not softened. band_hugs_floor asks whether the scores are piled on the bar; that needs the scores and the bar to come from the same regime. Where they do not, the honest answer is that this sample cannot say, plus the date after which one can — floor_moved_mid_window replaces band_hugs_floor for that arm and never accompanies it. A reader told a number is unavailable goes and gets one; a reader handed a qualified number uses it. NO MIGRATION. `actor` is Text with no CHECK precisely so a new kind of actor is not one — the model's own comment says so, and this is the case it anticipated. "release" joins "model" and "human". user_id is already nullable, which is right: no user did this, a release acts on every account that has not overridden the dial, and a row per user would both multiply and misattribute it. Both readers now take the newest of (this user's change, the release's). THE FIRST SIGHTING IS A BASELINE, written with old_value NULL. Nothing moved; the row exists so the next release has a predecessor. That null is load-bearing: floor_moves_since asks for old_value IS NOT NULL, so a fresh install's baseline does not silently retire the check on every new install. UI: the tuning history rendered actor as `human ? 'you' : 'Claude'`, so a release row would have told the operator that Claude moved a floor it never touched — the one failure the actor column exists to prevent. Three-way now, with an unknown value printing itself rather than guessing. Recorded at startup, inline and awaited. What #4181 cost three hours was concurrency — a background task racing the hook for the same pool. Sequential creates no contention, and this is twelve single-row reads. It must finish before serving because a readout served before the change was recorded is the exact answer this exists to stop giving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 01:17:37 -04:00
bvandeusenandClaude Opus 5	a7d736860f	fix(retrieval): the newest two rows are not the newest row of each dial (#4104) ... `current_settings` read a surface's history with one query and `limit(2)`, then keyed the rows by dial. That is only the same thing while both dials have moved equally often — and they do not. Floors get walked; budgets rarely move. Three floor changes and one budget change returns two floor rows, and the budget change vanishes. Under #4102 that cost a missing reason. Since #4104 it is worse: the dial then reports `source: "shipped"` — still on the value Scribe shipped — for a number somebody deliberately tuned. A wrong calibration answer, in the direction a reader has no cause to double-check. One query per dial, `limit(1)` each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-17 12:49:15 -04:00
bvandeusenandClaude Opus 5	aee24c9c1c	feat(retrieval): a tuned number carries the space it was measured in (#4104) ... Milestone 416 step 6. A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor on the install keeps applying while describing nothing — and nothing anywhere says so, because the scores simply come out different and the bar goes on cutting. `CHUNKER_VERSION` already solved this for documents: stamped per row, so the backfill re-embeds precisely what is stale. The same idea, applied to the numbers: - `calibration_stamp()` — embedding model + document shape, one definition. TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half moved, because they call for different responses. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (migration 0104), stamped on every write. Nullable and NOT backfilled — "unstamped" is the honest answer for a row written before this existed, and it reports as `stale: null`, never as fine. - `current_settings` reports calibration per dial: tuned rows from their event, untouched dials from the registry default's own stamp. - `retrieval_surfaces` and the Settings panel show the mismatch. The panel renders ONLY when something is stale, so seeing it at all is the signal. - `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to be inherited by the next model so that they don't have to recalibrate a lot": the raw cosine cannot cross models, but the PERCENTILE it represented can. Measure what fraction of a surface's logged calls the old floor admitted, re-score those queries under the current model, take the value admitting the same fraction. Dry run by default; applying writes an ordinary tuning event with the arithmetic in its reason. Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be, and #4102 measured the one case where the statistic and the correct action pointed opposite ways. The load-bearing test is an ABSENCE: no chat-model identifier may appear anywhere in the calibration path. Claude produces none of these scores, so a Claude upgrade must trigger nothing — a false alarm here teaches the operator to ignore the real one on the day bge-small becomes bge-base. Backup v17 carries both columns, unfilled on the way out and on the way back: a round trip must not turn "we don't know" into a stated fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-17 12:45:48 -04:00
bvandeusenandClaude Opus 5	6240652dce	feat(retrieval): the operator can see what was tuned, and every write is recorded (#4102) ... The other half of the bargain in milestone 416 step 4. The model moves these dials; this is what makes that reviewable rather than merely automatic. THE HOLE THIS CLOSES Every retrieval floor is an ordinary settings key, and `/api/settings` accepts any key at all. A floor written through it landed correctly and recorded nothing — a tuning history with holes in it, which is worse than no history because it reads as complete. So the generic endpoint now routes registry-owned keys through `set_dial` instead of writing them as plain rows. ROUTED, not refused: refusing would only work for callers that had been updated, while this way the form, a script, and an old client all leave the trail, and there is no version of "forgot to use the other endpoint". Clearing a control is written as an explicit set back to the shipped default, because the operator reverting something is the single most important move this history can record. `set_dial` now also refuses to record a no-op. The Settings form re-sends every field on every save, so without that one press of Save would write six rows saying the operator set six dials to the numbers they were already on — and a history nobody can skim is one nobody reads. WHAT THE OPERATOR GETS `/api/retrieval/surfaces`, `/surfaces/<name>` and `/tuning-history`, with `actor` fixed server-side rather than taken from the payload: a payload-supplied actor would let a model claim to be the operator, and "did I do this, or did the session?" is the first question this list is asked. In Settings: the five missing BUDGETS (until now only auto-inject had one, so the only control over a noisy surface was to raise its bar — which discards that surface's best candidates along with its worst), and a "What has been tuned" panel showing each change, who made it, and the reason given. The operator's own changes are marked. The MCP tool demands a reason; these endpoints do not. That asymmetry is deliberate and stated in routes/retrieval.py: the requirement exists to make the MODEL read the records before moving a number on someone else's behalf, and the operator is that someone — a mandatory justification box on every control would be friction charged to the one participant who owes no explanation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-17 11:36:22 -04:00
bvandeusenandClaude Opus 5	003bfd7a0a	fix(tests): two readers moved, and the settings guard now checks the registry (#4102) ... CI on 09b4845 caught both, and the second is an improvement rather than a repair. `test_services_reply_preferences` patched `reply_preferences.get_setting`, which the registry refactor removed — its floor and budget are resolved through `retrieval_surfaces` now. Its key-independence test also asserted the arm asks for exactly one key; it asks for two, because both of its numbers are its own since this step, so the assertion names both and adds a check that the registry and the module constant still agree about the floor key. They writing different keys is the failure where the Settings form saves one string and the arm reads another. `test_settings_defaults_agree` parsed `plugin_context.py` for a bare module-level float, and those constants now alias the registry. Rather than teach the regex about aliases, the six retrieval floors are keyed on their SURFACE NAME and read from the registry directly — which is strictly better for this guard: a surface name is also its telemetry source, so a row names the same arm the readout does, and a floor cannot be checked against a stale constant that happened to keep its old value. `PLAN_MATCH_DEFAULT_THRESHOLD` is not a push surface and keeps the older shape, with a note saying why. Added while there: `test_every_tunable_surface_has_a_control`, derived from the registry, so a seventh surface arrives as a failing test rather than as a number only the model can reach (rules 25, 27). Also lands the audit trail this step needs — `retrieval_tuning_events` (model + migration 0103) and `services/retrieval_tuning.py`. The tool layer on top is the next commit; the table is here because `reason` being REQUIRED is the whole guardrail, and the schema is where that starts. A number moved silently leaves nothing for the operator to review or disagree with, and the operator's decision is that the model moves these "9 times out of 10". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-17 11:20:47 -04:00