feat(telemetry): tell a ranker decline from a repeat before the observation window opens (#3497)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 37s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 28s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 37s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 28s
Making the rule arms log every call exposed a second ambiguity in the same
row. `result_count == 0` is two unrelated events wearing one number:
- the ranker found nothing above the bar — the only evidence a threshold is
set too high; and
- the ranker found only what this session had already been shown — which
says nothing whatever about the bar.
A long session excludes its way into the second, so the arm reads worse the
longer it runs correctly. Rows written now carry the ambiguity permanently,
which is why this lands before any watch period rather than after.
`retrieval_logs.suppressed_count` (0095, nullable) holds what the caller
dropped as already-shown. Both rule arms report it; they filter in Python and
always know. The note arms pass exclusions INTO semantic_search_notes and
never see what was dropped, so they store NULL.
THE NULL IS LOAD-BEARING. It means "not measured here", and the readout
renders it as `suppression: null` rather than a zeroed dict. Defaulting to 0
would let an unmeasured surface read as a perfectly clean one — the same
substitution of an artifact for a measurement that #3311 made. No backfill,
for the same reason: existing rows genuinely do not know.
`retrieval_telemetry`'s `sources` gains `suppression` with `measured_calls`,
`calls_with_suppression` and `zero_because_already_shown`; subtract the last
from `zero_result_calls` for the true ranker declines. The MCP tool docstring
says to read the two together and warns against reading the null as a zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -702,3 +702,65 @@ def test_neither_rule_arm_logs_its_call_behind_a_results_guard():
|
||||
"the pre-tool arm returns before logging its call — a surface with no "
|
||||
"rows at all cannot be told apart from a hook that never fired"
|
||||
)
|
||||
|
||||
|
||||
# ── Suppression: which zeros were the ranker, which were repeats (#3497) ──
|
||||
#
|
||||
# Making the call log unconditional exposed a second ambiguity in the same row.
|
||||
# A zero-result rule call is two unrelated events: the ranker found nothing
|
||||
# above the bar, or it found only what this session already held. Only the
|
||||
# first says anything about the threshold, and a long session excludes its way
|
||||
# into the second — so without the split, the arm looks worse the longer it
|
||||
# runs correctly.
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_write_path_arm_reports_what_the_session_already_held():
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
hits = [(0.71, fake_rule(id=156, title="A wait with no deadline is a bug")),
|
||||
(0.70, fake_rule(id=157, title="A loop re-arms in a finally"))]
|
||||
await _run_arm(hits, rec, retrieval_log=log, exclude_rule_ids=[156, 157])
|
||||
|
||||
row = next(c for c in log.call_args_list
|
||||
if c.kwargs.get("source") == "write_path_rule")
|
||||
assert row.kwargs["results"] == []
|
||||
assert row.kwargs["suppressed"] == 2, (
|
||||
"both hits were repeats, so this zero is not evidence about the bar"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_genuine_ranker_decline_reports_zero_suppression():
|
||||
"""Zero, not None. The arm filters in Python, so it always knows — and
|
||||
'measured none' has to stay distinguishable from 'cannot measure'."""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
await _run_arm([], rec, retrieval_log=log)
|
||||
|
||||
row = next(c for c in log.call_args_list
|
||||
if c.kwargs.get("source") == "write_path_rule")
|
||||
assert row.kwargs["suppressed"] == 0
|
||||
assert row.kwargs["suppressed"] is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_tool_arm_reports_suppression_too():
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
hits = [(0.75, fake_rule(id=161, title="Reach the forge through its MCP tools"))]
|
||||
await _run_tool_arm(hits, rec, retrieval_log=log, exclude_rule_ids=[161])
|
||||
|
||||
assert log.call_args.kwargs["results"] == []
|
||||
assert log.call_args.kwargs["suppressed"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_shown_hit_is_not_counted_as_suppressed():
|
||||
"""The obvious inverse, worth pinning: `suppressed` counts what was DROPPED,
|
||||
not what came back. Off by one here and every zero row reads as a repeat."""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
hits = [(0.75, fake_rule(id=161, title="Reach the forge through its MCP tools")),
|
||||
(0.70, fake_rule(id=12, title="Don't run a local stack unless asked"))]
|
||||
out = await _run_tool_arm(hits, rec, retrieval_log=log, exclude_rule_ids=[12])
|
||||
|
||||
assert out["rule_ids"] == [161]
|
||||
assert log.call_args.kwargs["suppressed"] == 1
|
||||
assert len(log.call_args.kwargs["results"]) == 1
|
||||
|
||||
Reference in New Issue
Block a user