4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	f8e53c1c35	fix(telemetry): a warning fired on an arm whose decline rate is arithmetic, not evidence (#4232) ... Found by reading a live `retrieval_telemetry` readout after milestone 419 deployed, not by inspection. The readout said: cannot_decline / report_preference — "45 calls, 0 of them returned nothing. An arm that fires unasked has to be able to say nothing; this one never has. Check that it applies its floor at all." And printed, beside it, that arm's band: p10 = p50 = p90 = min = max = 0.791. FIVE IDENTICAL PERCENTILES IS THE TELL. That is not a ranking, it is one record at one score on every call — because `report_preference` searches a fixed string (`reply_preferences.COMPLETION_QUERY`, a module constant, and deliberately so). For a fixed query against a stable corpus the top score is a CONSTANT, so the arm's decline rate is 0% or 100% and never in between; which of the two it is depends only on where the bar sits relative to that one number. "Never returned nothing" is therefore arithmetic, not evidence, and the warning's own remedy — check whether it applies a floor — cannot be answered from it. The arm already knew this about itself; the warning did not: "a fixed query makes this arm's score a constant and a floor a hair above it produces a dead arm no amount of traffic will ever reveal" — services/reply_preferences.py THIS CLASS OF BUG ALREADY HAS A GUARD, which is the argument for the shape of the fix. `Point.logs_unconditionally` exists because of #3497: both rule arms once logged only their hits, so their zero count was structurally 0 and this same warning would have fired on a LOGGING property while sending the reader to move a threshold that was never involved. This is that one step over — a QUERY-SHAPE property — and gets the same treatment: a declared field on `Point`, and exclusion rather than trust. AND THE WARNING THAT WOULD BE INFORMATIVE HERE DID NOT EXIST. For a fixed-query arm the dangerous state is the mirror image: every call empty, meaning the bar is above the constant and no further traffic will ever move it. The arm is off rather than quiet, and nothing in the readout said so — `expects_traffic` covers an arm with NO calls, not one with calls and a 100% decline rate. That state is real and reached: `report_preference` once logged 69 consecutive declines at 0.0006 under its bar. So `fixed_query_never_clears` sends the reader to `near_miss_samples` and not to the dial — because that incident is also the one where the statistic and the correct action pointed opposite ways. Every percentile said lower the floor; opening the refused record showed it was rule 77 arriving as a false positive, and lowering it would have delivered that rule on every completion report ever written. Guards in tests/test_retrieval_warnings.py, including the falsifier that matters most here: `cannot_decline` must still fire on an arm whose query varies, or this change is a disabled check wearing a narrowed one's clothes. Both boundaries tested from both sides, per that module's own standard. The new code is documented in the `retrieval_telemetry` tool docstring beside the others (rule 33) — an undocumented code in a readout is a reader meeting a verdict with no way to disagree with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 08:05:52 -04:00
bvandeusenandClaude Opus 5	512d0326a0	fix(telemetry): a floor that moved inside the window makes the band check a comparison of two populations (#4225) ... `retrieval_telemetry(days=30)` reported, for write_path_rule: "the weakest tenth of what this arm returns scores 0.6984, only -0.0216 above its floor of 0.72" A negative distance above something. The tenth percentile of what an arm RETURNED cannot sit below the floor that gates what it may return — not inside one population. MEASURED CAUSE. write_path_rule's floor was 0.68 until 2026-09-02, when 2385100 (#3318) raised the shipped default to 0.72. The window opened 2026-08-22, so six days of it are calls made under the old bar; top_score.min for the surface is exactly 0.68, the old bar still in the sample. AND THE CHANGE LEFT NO TRACE THE READOUT COULD SEE. retrieval_tuning_events records dial turns — a person or a model choosing a number. It was silent about the other way a floor moves: somebody edits floor_default and ships it. retrieval_tuning_history returned {"events": []} and retrieval_surfaces said last_change: {}, source: "shipped". All true, and all of it silent about a floor that had in fact moved. THE RAISE ANNOUNCED ITSELF. A LOWERED FLOOR WOULD NOT: the gap comes out comfortably positive and reads as a clean bill of health on a sample that half predates the bar being judged. Both directions are now pinned. So the check is SUSPENDED, not softened. band_hugs_floor asks whether the scores are piled on the bar; that needs the scores and the bar to come from the same regime. Where they do not, the honest answer is that this sample cannot say, plus the date after which one can — floor_moved_mid_window replaces band_hugs_floor for that arm and never accompanies it. A reader told a number is unavailable goes and gets one; a reader handed a qualified number uses it. NO MIGRATION. `actor` is Text with no CHECK precisely so a new kind of actor is not one — the model's own comment says so, and this is the case it anticipated. "release" joins "model" and "human". user_id is already nullable, which is right: no user did this, a release acts on every account that has not overridden the dial, and a row per user would both multiply and misattribute it. Both readers now take the newest of (this user's change, the release's). THE FIRST SIGHTING IS A BASELINE, written with old_value NULL. Nothing moved; the row exists so the next release has a predecessor. That null is load-bearing: floor_moves_since asks for old_value IS NOT NULL, so a fresh install's baseline does not silently retire the check on every new install. UI: the tuning history rendered actor as `human ? 'you' : 'Claude'`, so a release row would have told the operator that Claude moved a floor it never touched — the one failure the actor column exists to prevent. Three-way now, with an unknown value printing itself rather than guessing. Recorded at startup, inline and awaited. What #4181 cost three hours was concurrency — a background task racing the hook for the same pool. Sequential creates no contention, and this is twelve single-row reads. It must finish before serving because a readout served before the change was recorded is the exact answer this exists to stop giving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 01:17:37 -04:00
bvandeusenandClaude Opus 5	bb8013928f	feat(telemetry): the readout names rules that were opened and changed nothing (#4213) ... Milestone 419 step 2. Step 1 made an outcome recordable; this makes it readable. `retrieval_summary`'s rule block gains `applied`, `departed` and `distinct_rules_acted`, and `_compute_warnings` gains two codes. TWO CODES, NOT ONE WITH A ZERO IN IT. `read_and_unacted` reports rules that were opened and left no outcome, against the ones that did. It only fires once outcomes exist anywhere in the window, because a window with none cannot tell "every rule was ignored" from "nothing calls `rule_outcome` yet" — and on every install the day this ships, the truth is the second. Claiming the first there would be #3311's failure exactly: a statistic that could not vary being read as a fact about the corpus. The cold case gets its own code, `outcomes_never_recorded`, whose prose says in as many words that it does NOT mean the rules were ignored. `applied` AND `departed` ARE NOT SUMMED. A departure carries the reason the agent gave and is evidence about the RULE; an application is evidence about the agent. Folded together they would say only "an outcome exists", which is true of both and useful about neither. `distinct_rules_acted` counts either, because for the unacted arithmetic the distinction does not matter. An outcome is not a pull. The fold branches on OUTCOMES first and never routes an outcome through the surfaced/ambient split: `source` on an outcome row names the door the outcome came through, not a ranker, so the ambient distinction has nothing to say about it. An integration test holds that line — if an outcome leaked into the pull counters the silently-unchanged rule would vanish into a compliant-looking total, which is the confusion #4212 was opened to end. Verified by lifting the shipped `_compute_warnings` out of source with `ast` and exercising it against the six populations the new tests assert: cold instrument, warm instrument, departures-only, full compliance, nothing opened, and a failed read. The integration tests for the new counts run against real Postgres in CI — count(distinct) with an IN over an unconstrained column is a SQL shape a mock would agree with whatever it did, which is what #2663 was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 00:13:45 -04:00
bvandeusenandClaude Opus 5	e2c3a5c2b5	feat(telemetry): retrieval_telemetry says what is wrong (#3431) ... The tool returned distributions and left the reading to the caller, so every readout was the same four checks done by hand — #3430's baseline, #3835's rule near-misses, the #1038 rerank gate. Mechanical, and therefore forgettable. Tonight's acceptance pass on #3898 was the case for doing this. Reading it by hand meant catching that two surfaces had `covers_window: false`, that `prompt_rule`'s floor had moved three times inside the window (which made the readout self-contradictory: deliveries at 0.622 beside refusals at 0.7199), and that 15 of 20 near-misses were one record against text no operator wrote. Miss any of those and the obvious conclusion was "the bar is too tight" — a floor change that would have injected one preference into every notification. `warnings` is always present and empty when clean, so its emptiness is an answer rather than a gap. Each entry carries the numbers that produced it: "345 calls, 0 declined" is the analysis, "check write_path_rule" is an instruction to redo it. Five codes — cannot_decline, band_hugs_floor, no_duration, surfaced_never_pulled, unregistered_source. cannot_decline has three guards, each a bug it would otherwise cause. Asked surfaces are exempt (a search returning a list every time is working). An arm not known to log unconditionally is exempt — that is #3497 exactly, where both rule arms recorded only their hits, so a decline count of zero was a LOGGING defect and this warning would have sent the reader to a threshold that was never involved. Unregistered sources get numbers but no verdict. `silent_surfaces` is the half the rows cannot show: an arm that emitted nothing is invisible to every row-based check and looks exactly like an arm that does not exist. It is driven by a new declared registry, `retrieval_registry.POINTS` — deliberately NOT `retrieval_surfaces.SURFACES`, which answers "what can be tuned" and excludes the reserved slots because a budget of 1 is their feature. This answers "what can be measured", and the reserved slots belong in it precisely because they are judgeable without being tunable. A test asserts the two cannot drift apart. The registry test derives sources from the call sites with `ast`, not grep, and the difference is not theoretical: `wide_net` and `report_preference` reach their recorder as `source=SOURCE` through a module constant, so a grep for `source="` is blind to both — the narrowing #3191 warns about. Three sites pass `source` as a variable and are declared in FAN_OUT_SITES; the test pins those sites but not the values they can pass, which is why the `unregistered_source` warning exists to catch the rest at first fire. Thresholds are settings (rule 25) defaulted so a fresh install with almost no data produces no warnings at all (rule 115) — a new user's first readout naming five broken things would be describing the emptiness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-20 20:24:56 -04:00