Milestone 379 closes: the readout can now say what it did not measure #144

Merged

bvandeusen merged 3 commits from dev into main 2026-09-08 16:06:15 -04:00

Conversation 0 Commits 3 Files Changed 9

+519 -49

bvandeusen commented 2026-09-08 16:06:09 -04:00

Owner

Copy Link

Copy Source

Finishes milestone #379. Every change here is to the readout that made a phantom 64% write loss believable — not to the write path, which was never broken.

#3668 — the identity that falsified the milestone is now a guard

Both rule arms build one fresh list and hand it to two recorders in one function, so the call log and the surfacing log cannot disagree about a single call. Parametrized over both arms × three exclusion states, asserting ids rather than counts — equal counts drawn from different lists is a real way for this to break.

Falsified rather than argued (rule 167, and rule 10 makes CI the only executor). 5e19a1b deliberately broke results=fresh → results=hits in both arms; run 6046 failed exactly the predicted four cases and passed the two nothing-held ones. That last part is the finding: with no exclusions both recorders see the same list however wrongly they are wired, so a guard built only from the obvious case would have read as coverage while catching nothing. 277aea5 reverts the break; plugin_context.py is byte-identical to its pre-falsification state.

Deliberately not built: the readout-level self-check the task also proposed. cleared_threshold counted calls, surfaced counts rules, and those coincide only while RULEHINT_LIMIT is 1 — it would fire on a healthy system the moment the limit rises.

#3670 — the bar can only be judged from what it rejected

cleared_threshold was documented as the number to read first. It was a tautology: the search applies the bar before returning, so every returned result cleared it by construction and a call with no results has no score to compare. It was calls - zero_result_calls under a name that promised a second opinion, and zero + cleared == calls held on all nineteen readings ever taken.

Replaced, not just removed. The question the table exists for is only answerable from the calls that returned nothing — a 0.72 bar turning away a stream of 0.71s is set too high; the same bar turning away 0.30s is working. Both render as a zero-result call today, because the losing score is discarded inside the search.

So both searches now rank without the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one. Only the losers' visibility changes. report carries the score out without changing what a search returns: eight of the eleven call sites want hits and nothing else.

near_misses cannot degenerate the way its predecessor did, for a structural reason rather than a naming one — it is measured on the calls the bar rejected, using a score the bar never saw.

Migration 0096

Adds retrieval_logs.best_available_score, nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence.

near_misses reads null everywhere until rows accumulate after the deploy. That is correct, and it is what the null is for.

The risk worth naming

The new aggregate is a percentile_cont ... WITHIN GROUP over a CASE, in the grouped query that already carries four other CASEs. That is one step from the shape that produced #2663 — a query Postgres rejected, swallowed by this module's broad except, every counter reading zero in production while the writes landed and the mocked tests passed. The failure mode is silence, so it carries an integration guard whose first assertion is read_failed is False.

CI green on e7c1af3 (run 6057), all six lanes including integration.

🤖 Generated with Claude Code

https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

Finishes milestone #379. Every change here is to the *readout* that made a phantom 64% write loss believable — not to the write path, which was never broken. ## #3668 — the identity that falsified the milestone is now a guard Both rule arms build one `fresh` list and hand it to two recorders in one function, so the call log and the surfacing log cannot disagree about a single call. Parametrized over both arms × three exclusion states, asserting **ids rather than counts** — equal counts drawn from different lists is a real way for this to break. **Falsified rather than argued** (rule 167, and rule 10 makes CI the only executor). `5e19a1b` deliberately broke `results=fresh` → `results=hits` in both arms; run 6046 failed exactly the predicted four cases and **passed** the two `nothing-held` ones. That last part is the finding: with no exclusions both recorders see the same list however wrongly they are wired, so a guard built only from the obvious case would have read as coverage while catching nothing. `277aea5` reverts the break; `plugin_context.py` is byte-identical to its pre-falsification state. Deliberately **not** built: the readout-level self-check the task also proposed. `cleared_threshold` counted calls, `surfaced` counts rules, and those coincide only while `RULEHINT_LIMIT` is 1 — it would fire on a healthy system the moment the limit rises. ## #3670 — the bar can only be judged from what it rejected `cleared_threshold` was documented as the number to read first. It was a tautology: the search applies the bar before returning, so every returned result cleared it by construction and a call with no results has no score to compare. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen readings ever taken. **Replaced, not just removed.** The question the table exists for is only answerable from the calls that returned *nothing* — a 0.72 bar turning away a stream of 0.71s is set too high; the same bar turning away 0.30s is working. Both render as a zero-result call today, because the losing score is discarded inside the search. So both searches now rank **without** the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one. Only the losers' visibility changes. `report` carries the score out without changing what a search *returns*: eight of the eleven call sites want hits and nothing else. `near_misses` cannot degenerate the way its predecessor did, for a structural reason rather than a naming one — it is measured on the calls the bar rejected, using a score the bar never saw. ## Migration 0096 Adds `retrieval_logs.best_available_score`, nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence. **`near_misses` reads null everywhere until rows accumulate after the deploy.** That is correct, and it is what the null is for. ## The risk worth naming The new aggregate is a `percentile_cont ... WITHIN GROUP` over a `CASE`, in the grouped query that already carries four other CASEs. That is one step from the shape that produced #2663 — a query Postgres rejected, swallowed by this module's broad `except`, every counter reading zero in production while the writes landed and the mocked tests passed. The failure mode is **silence**, so it carries an integration guard whose first assertion is `read_failed is False`. CI green on `e7c1af3` (run 6057), all six lanes including integration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

bvandeusen added 3 commits 2026-09-08 16:06:09 -04:00

test(telemetry): FALSIFICATION — prove the identity guard can fail (#3668) ... 5e19a1b028

Deliberately broken, reverted in the next commit. Rule 167 requires every
guard be falsified against the regression it names before it is trusted,
and rule 10 puts CI as the only place that can run it — so the failure has
to be made to happen here rather than argued for.

The guard: both rule arms feed one `fresh` list to two recorders, so the
call log and the surfacing log cannot disagree about what one call showed.

The regression: `results=fresh` becomes `results=hits` in both arms, so
the call log counts what the ranker found while the surfacing log counts
what was shown. That is not a hypothetical shape. It is exactly the
divergence that would make a correct system report a lost write when the
two tables are later compared in aggregate — the reading that scoped this
milestone at five steps against a defect that did not exist.

Expected red: the one-already-held and all-already-held cases on both
arms. The nothing-held case must still PASS — with no exclusions both
recorders see the same list however wrongly they are wired, which is why
it could never have been the discriminating case.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

test(telemetry): pin the identity that falsified this milestone (#3668) ... 277aea58e4

Reverts the deliberate break from 5e19a1b. `plugin_context.py` is now
byte-identical to before it; the only change against that baseline is the
guard itself.

FALSIFIED, not argued (rule 167). CI run 6046 with `results=hits` in both
arms failed exactly the predicted four cases —

    [one-already-held-write_path]   FAILED
    [one-already-held-pre_tool]     FAILED
    [all-already-held-write_path]   FAILED
    [all-already-held-pre_tool]     FAILED
    [nothing-held-*]                passed

— and the nothing-held cases passing is the point, not a gap: with no
exclusions both recorders see the same list however wrongly they are
wired, so that case can never discriminate and a guard built only from it
would read as coverage while catching nothing.

WHAT IS PINNED. Both arms build one `fresh` list and hand it to two
recorders in one function, so the call log and the surfacing log cannot
disagree about what a single call showed. Ids, not counts: equal counts
drawn from different lists is a real way for this to break, and a count
comparison would call it agreement.

Three hits, where production returns at most one. The identity holds at
any limit because both recorders read the same list, and stating it that
way survives RULEHINT_LIMIT moving again — it has moved once already
(2 → 1, 2385100), and that move is half of why the original
reconstruction misread its own numbers.

NOT DONE, deliberately: the readout-level self-check the task also
proposed. `cleared_threshold` counts CALLS that beat the bar while
`surfaced` counts RULES, so that identity holds only while the limit is
1 — it would fire on a healthy system the moment the limit rises. The
arm-level form has no such coupling, which is why the guard lives here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

fix(telemetry): the bar can only be judged from what it rejected (#3670) ... e7c1af32a0

`cleared_threshold` was documented as the number to read first. It was a
tautology. The search applies the threshold before returning, so every
returned result cleared it by construction and a call with no results has
no top_score to compare — the condition was true exactly when
`result_count > 0`. It was `calls - zero_result_calls` under a name that
promised a second opinion, and `zero + cleared == calls` held on all
nineteen source/window readings ever taken, today's live seven included.

The reading procedure built on it asked the reader to compare a number
with itself, and a threshold change was unobservable through it: raise the
bar and both numbers move together, so the field could never show a bar
set too high.

REPLACED, NOT JUST REMOVED. The question the table exists to answer is
whether the bar is in the right place, and that is only answerable from
the calls that returned NOTHING: how close did the best rejected candidate
come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair;
the same bar turning away 0.30s is working. Both render as a zero-result
call today and nothing separates them, because the losing score is
discarded inside the search.

So both searches now rank WITHOUT the bar and apply it in Python. The
qualifying set is provably identical — rows arrive ordered by distance, so
every above-bar row sorts ahead of every below-bar one, and an over-fetch
that returned N above-bar rows returns the same N plus some losers. What
changes is that the losers are visible instead of dropped in the query.
`report` carries the score out without changing what a search RETURNS:
eight of eleven call sites want hits and nothing else.

New column (migration 0096), nullable and unbackfilled. A row written
before this genuinely does not know, and a 0.0 would read as "the corpus
held nothing remotely relevant" — a claim invented out of a caller's
silence, which is the substitution this whole milestone corrects.

The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step
from the shape that produced #2663, where a rejected query was swallowed
by the broad except and every counter read zero. It carries an integration
guard for that reason: only real Postgres can say it parses, and the
symptom of failure is silence.

Also adds a guard that no int field in a bucket equals
`calls - zero_result_calls`. That identity is what `cleared_threshold`
satisfied for its whole life, and it survived because it had its own name
and nobody added the two numbers beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

bvandeusen merged commit 53df6742cb into main 2026-09-08 16:06:15 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#144