Milestone 379 closes: the readout can now say what it did not measure #144

Merged

bvandeusen merged 3 commits from dev into main 2026-09-08 16:06:15 -04:00

Conversation 0 Commits 3 Files Changed 9

+519 -49

3 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	e7c1af32a0	fix(telemetry): the bar can only be judged from what it rejected (#3670) ... `cleared_threshold` was documented as the number to read first. It was a tautology. The search applies the threshold before returning, so every returned result cleared it by construction and a call with no results has no top_score to compare — the condition was true exactly when `result_count > 0`. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen source/window readings ever taken, today's live seven included. The reading procedure built on it asked the reader to compare a number with itself, and a threshold change was unobservable through it: raise the bar and both numbers move together, so the field could never show a bar set too high. REPLACED, NOT JUST REMOVED. The question the table exists to answer is whether the bar is in the right place, and that is only answerable from the calls that returned NOTHING: how close did the best rejected candidate come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair; the same bar turning away 0.30s is working. Both render as a zero-result call today and nothing separates them, because the losing score is discarded inside the search. So both searches now rank WITHOUT the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one, and an over-fetch that returned N above-bar rows returns the same N plus some losers. What changes is that the losers are visible instead of dropped in the query. `report` carries the score out without changing what a search RETURNS: eight of eleven call sites want hits and nothing else. New column (migration 0096), nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence, which is the substitution this whole milestone corrects. The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step from the shape that produced #2663, where a rejected query was swallowed by the broad except and every counter read zero. It carries an integration guard for that reason: only real Postgres can say it parses, and the symptom of failure is silence. Also adds a guard that no int field in a bucket equals `calls - zero_result_calls`. That identity is what `cleared_threshold` satisfied for its whole life, and it survived because it had its own name and nobody added the two numbers beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:50:05 -04:00
bvandeusenandClaude Opus 5	277aea58e4	test(telemetry): pin the identity that falsified this milestone (#3668) ... Reverts the deliberate break from 5e19a1b. `plugin_context.py` is now byte-identical to before it; the only change against that baseline is the guard itself. FALSIFIED, not argued (rule 167). CI run 6046 with `results=hits` in both arms failed exactly the predicted four cases — [one-already-held-write_path] FAILED [one-already-held-pre_tool] FAILED [all-already-held-write_path] FAILED [all-already-held-pre_tool] FAILED [nothing-held-*] passed — and the nothing-held cases passing is the point, not a gap: with no exclusions both recorders see the same list however wrongly they are wired, so that case can never discriminate and a guard built only from it would read as coverage while catching nothing. WHAT IS PINNED. Both arms build one `fresh` list and hand it to two recorders in one function, so the call log and the surfacing log cannot disagree about what a single call showed. Ids, not counts: equal counts drawn from different lists is a real way for this to break, and a count comparison would call it agreement. Three hits, where production returns at most one. The identity holds at any limit because both recorders read the same list, and stating it that way survives RULEHINT_LIMIT moving again — it has moved once already (2 → 1, 2385100), and that move is half of why the original reconstruction misread its own numbers. NOT DONE, deliberately: the readout-level self-check the task also proposed. `cleared_threshold` counts CALLS that beat the bar while `surfaced` counts RULES, so that identity holds only while the limit is 1 — it would fire on a healthy system the moment the limit rises. The arm-level form has no such coupling, which is why the guard lives here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:36:03 -04:00
bvandeusenandClaude Opus 5	5e19a1b028	test(telemetry): FALSIFICATION — prove the identity guard can fail (#3668) ... Deliberately broken, reverted in the next commit. Rule 167 requires every guard be falsified against the regression it names before it is trusted, and rule 10 puts CI as the only place that can run it — so the failure has to be made to happen here rather than argued for. The guard: both rule arms feed one `fresh` list to two recorders, so the call log and the surfacing log cannot disagree about what one call showed. The regression: `results=fresh` becomes `results=hits` in both arms, so the call log counts what the ranker found while the surfacing log counts what was shown. That is not a hypothetical shape. It is exactly the divergence that would make a correct system report a lost write when the two tables are later compared in aggregate — the reading that scoped this milestone at five steps against a defect that did not exist. Expected red: the one-already-held and all-already-held cases on both arms. The nothing-held case must still PASS — with no exclusions both recorders see the same list however wrongly they are wired, which is why it could never have been the discriminating case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:31:49 -04:00