write_path stops reporting records it withheld itself as near misses #146
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: bvandeusen/FabledScribe#146
Reference in New Issue
Block a user
Follow-up to the previous PR, found by verifying it on live data — the check that fix was built to make possible.
The reading that exposed it
A "rejection" that outscored every acceptance — and not one outlier, since the p90 (0.7521) is above the bar too, across 47 measured calls.
Why the previous fix missed it
That fix keyed on
suppressed_count, and its NULL branch was justified as "null means the caller passed its exclusions INTO the search, so the reported score is already post-exclusion and cannot be contaminated."True of
auto_injectandreuse_slot— both log the raw search output and do any Python filtering afterwards, so their count and score agree. False ofwrite_path, the one note arm that filters twice:exclude_idstakesseen - pulled_seeninto the search, but the pulled-and-seen ids stay in the query deliberately (this arm's query doubles as the resemblance test thatresemblesneeds) and are dropped afterwards in Python. The row therefore carries a post-filter count beside a pre-filter score.The justification was asserted as though all three arms had been checked. It was reasoned from the pattern of two.
The fix, and why not the obvious one
best_availableis reported only when this arm's own filter removed nothing — null otherwise, meaning "not measured on this call, because the bar was not the only thing that turned something away". Most calls still report.Not the suppression column, which is what rescued the rule arms: this arm's count would be partial, covering the drops it makes in Python but not the ones
exclude_idsalready made inside the search. A partial number under a name that reads as complete is the substitution this whole milestone exists to stop.Null also composes with the existing
best_available_score IS NOT NULLclause, so the aggregate predicate needs no widening.Both directions asserted
Without the second test, setting the field to null unconditionally passes the first while silently deleting the measurement #3670 was built for.
CI green on
277f5df(run 6138), all six lanes.🤖 Generated with Claude Code
https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
Found by verifying the previous fix on live data — the check that fix was meant to make possible. write_path near_misses.max 0.822 p90 0.7521 top_score.min 0.6857 so the bar is at or below this A "rejection" that outscored every acceptance, and not one outlier: the p90 is above the bar too. #3739's fix keyed on `suppressed_count`, and I justified its NULL branch as "null means the caller passed its exclusions INTO the search, so the score is already post-exclusion". That holds for auto_inject and reuse_slot, both of which log the RAW search output and do their Python filtering after. It does not hold for write_path, the one note arm that filters TWICE: `exclude_ids` takes `seen - pulled_seen` into the search, but the pulled-and-seen ids stay in the query on purpose — the arm's query doubles as the resemblance test — and are dropped afterwards in Python. So the row carries a POST-filter count beside a PRE-filter score. The suppression column cannot rescue it the way it does for the rule arms. This arm's count would be PARTIAL — covering the drops made here and not the ones `exclude_ids` made inside the search — and a partial number under a name that reads as complete is the substitution this milestone exists to stop. So it reports null whenever its own filter removed anything: not measured on this call, because the bar was not the only thing that turned something away. Calls that withheld nothing keep reporting, which is most of them. Both directions are asserted. Without the second test, setting the field to null unconditionally would pass the first while deleting the measurement #3670 was built for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ