feat(rules): the rule arms stop filtering the corpus to one tier (#3702)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 33s
CI & Build / TypeScript typecheck (push) Successful in 1m3s
CI & Build / Python tests (push) Successful in 1m7s
CI & Build / Build & push image (push) Successful in 1m29s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 33s
CI & Build / TypeScript typecheck (push) Successful in 1m3s
CI & Build / Python tests (push) Successful in 1m7s
CI & Build / Build & push image (push) Successful in 1m29s
Both arms passed tier="conditional", on the reasoning that an always-on
rule is already in the session so re-surfacing it is pure noise. That
conflates two different things:
PRESENT IN CONTEXT the rule was delivered at session start
SALIENT AT THE MOMENT the rule is in front of the reader when the action
it governs is about to be taken
A rule handed over in a list at turn zero is present while a session writes
a config value three hundred turns later. It is not surfaced. So the filter
did not skip a redundant hint — it made a whole class of rules permanently
ineligible for the only mechanism that puts a rule in front of an agent AT
the moment, and the more important a rule is, the likelier it sat in that
class.
Underneath, the filter was doing the THRESHOLD's job. Whether a rule belongs
in a hint is a relevance question and a similarity bar is the control for
relevance. A categorical exclusion standing in for a relevance judgment
cannot be tuned, cannot be measured, and cannot be wrong in a way anybody
notices.
MEASURED, NOT SETTLED. The old comment's fear is real: a hint that fires on
every write and says obvious things teaches the reader to skip the block. It
had simply never been checked, and retrieval_logs already records the scores
to check it with. Rules clearing often and high means the fear was justified
and the BAR is the work; rules clearing rarely in a thin band means relevance
was always sufficient.
Only eligibility moved. The bar stays at 0.72 and k stays at 1, so the
resulting distribution has one cause — and k=1 bounds the blast radius: a
wider pool can change which rule surfaces and how often, never how long a
single hint gets.
The threshold rationale's first premise ("the eligible corpus is TINY —
tier=conditional only") is updated rather than deleted: a larger pool makes
clearing the bar mean MORE, so that argument weakened, and the bar was left
alone anyway rather than move two variables at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -160,7 +160,15 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
|
||||
kw = search.await_args.kwargs
|
||||
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
|
||||
assert kw["limit"] == pc.RULEHINT_LIMIT
|
||||
assert kw["tier"] == "conditional"
|
||||
# NO tier filter (#3702). The arms search every rule the caller owns,
|
||||
# because "already in the session" is not the same as "in front of the
|
||||
# reader at the moment it applies" — and relevance is the threshold's
|
||||
# job, not a category's. If this assertion is failing because a tier
|
||||
# argument came back, read the block above RULEHINT_LIMIT first: the
|
||||
# filter may legitimately return, but only carrying a measured reason.
|
||||
assert "tier" not in kw or kw["tier"] is None, (
|
||||
"the arm is filtering the rule corpus by tier again"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user