feat(rules): a repeat is referenced, not withheld (#3750, #3752)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 40s
CI & Build / Python tests (push) Successful in 1m9s
CI & Build / Build & push image (push) Successful in 37s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 11s
CI & Build / integration (push) Successful in 40s
CI & Build / Python tests (push) Successful in 1m9s
CI & Build / Build & push image (push) Successful in 37s
Both arms used to drop a hit already on the session's exclusion ledger and emit nothing. That is correct only while the session still HOLDS what it was told, and a compaction breaks exactly that: the earlier injection is summarized away while the id stays on the ledger, leaving the rule absent from context AND unreachable for the rest of the session. #3749 closed the compaction half by clearing the ledger; this closes the ordinary half, where a session simply stops holding a line it read an hour ago. Only ONE CLAUSE of the existing line is false on a repeat — the claim that the rule is not in the session's loaded set — so only that clause changes. The fresh line is byte-identical to what it was. Both tails now come from one `_rule_hint_line`. The arms phrase their heads differently on purpose; everything after must not differ, and #3497's history is that the pre-tool arm inherited a defect by being modelled on its sibling rather than sharing with it. THE BUDGET DECISION, recorded at RULEHINT_LIMIT. A repeat competes for the single slot on rank alone: nothing is fetched behind it, and it never rides alongside as a second line. Promoting a fresh rule past a better-ranked repeat would reinstate the withholding one rank deeper, and a second line is the one thing the limit exists to forbid. The consequence is deliberate — a rule that keeps ranking first for a recurring situation keeps being referenced, and its decay belongs to exclusion ageing (#3751), not to a first-place rule being demoted for having won before. THE TELEMETRY, decided before shipping rather than after a number moved (#3752): nothing changes. A reference is a RENDERING decision, not a retrieval outcome. `results` stays `fresh`, `suppressed` stays len(hits) - len(fresh), and `record_rule_surfaced` still counts only what the arm freshly chose. This matters more than it reads: the naive implementation drops the `fresh` filter and takes suppressed_count to zero everywhere — and #3739's near-miss fix identifies repeat-caused zeros by `suppressed_count > 0`, so the contamination corrected on 2026-09-08 would return by a different route, in the same field, with the fix still in the code and no longer working. A test asserts the counters as unmoved, because "nothing changed" is only worth something if it is checkable. Also corrects two comments that outlived #3702 — both arms still claimed CONDITIONAL ONLY while the module-level note above RULEHINT_LIMIT said the opposite, in the exact code this change edits. GUARDS - an already-held hit produces a rule line at all (the regression), asserted on `get_rule(<id>)` rather than a truthy context: the write-path arm fills its context from four other sources, so truthiness passes under the OLD behaviour and pins nothing there. - the two tails are distinguishable, each excludes the other, and neither injects the rule statement — the budget claim, both branches. - the counters are unmoved, per #3752. - test_a_rule_the_session_already_holds_is_not_re_offered asserted the old contract (`"161" not in context`). Repinned rather than deleted: the telemetry half of what it protected still holds. Repinned the #3497 log-placement guard on structure (rule 167). It read `body.index("if not fresh:")` — a local variable NAME, not the property. This change renames that guard to `if not hits:`, so the old assertion would have raised ValueError and reported #3497 as back while the arm was entirely correct. Now walks the AST for the first early return after the search and asserts the call row is written before it. Falsified both ways: it fails on the #3497 mutation, and it refuses to pass when no early return exists at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -149,6 +149,37 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
|
||||
# adds a way to misconfigure the surface (rule 25 cuts both ways).
|
||||
RULEHINT_LIMIT = 1
|
||||
|
||||
# AND A REPEAT COMPETES FOR THAT ONE SLOT ON RANK ALONE (#3750).
|
||||
#
|
||||
# Since #3750 a hit already on the session's exclusion ledger is RENDERED
|
||||
# rather than dropped, which raises a question the old behaviour never had to
|
||||
# answer: when the top-ranked hit is one the session has already seen, does it
|
||||
# take the slot, or step aside for a fresh rule behind it?
|
||||
#
|
||||
# It takes the slot, and nothing is fetched behind it. Two reasons.
|
||||
#
|
||||
# RANK IS THE ANSWER TO "WHAT IS RELEVANT NOW". If the repeat scores 0.85 and
|
||||
# the best fresh candidate 0.73, the repeat is the better match for the action
|
||||
# actually being taken. Overfetching in order to promote the fresh one past it
|
||||
# would reinstate exactly the withholding this milestone exists to remove, one
|
||||
# rank deeper and harder to see — recency is not a reason to show a worse
|
||||
# match, and "you have seen this" is not the same claim as "you are holding
|
||||
# this".
|
||||
#
|
||||
# AND A SECOND LINE IS THE ONE THING THE LIMIT ABOVE FORBIDS. Letting a repeat
|
||||
# ride alongside a fresh rule means two hint lines, and the paragraph above is
|
||||
# entirely about why a fourth voice that speaks twice is where a reader stops
|
||||
# reading. A reference costs the same ~40 tokens as a first surfacing, so
|
||||
# "it is only a short extra line" is not available as an argument: the budget
|
||||
# is one line because of what a second line does to the whole hint, not
|
||||
# because of what it costs.
|
||||
#
|
||||
# The consequence is deliberate and worth naming: a rule that keeps ranking
|
||||
# first for a recurring situation keeps being referenced, every time the
|
||||
# situation recurs. That is the intended behaviour — the situation recurring
|
||||
# IS the trigger — and its decay belongs to exclusion ageing (#3751), not to
|
||||
# a rule that ranks first being quietly demoted for having won before.
|
||||
|
||||
# WHY THE ARMS NO LONGER FILTER TO ONE TIER (#3702).
|
||||
#
|
||||
# Both arms used to pass `tier="conditional"`, on the reasoning that an
|
||||
@@ -827,6 +858,52 @@ async def get_writepath_config(user_id: int) -> dict:
|
||||
"rule_threshold": rule_threshold,
|
||||
}
|
||||
|
||||
def _rule_hint_line(rule, *, where: str, seen: bool) -> str:
|
||||
"""One rule hint line — both arms, both tails (#3750).
|
||||
|
||||
ONE FUNCTION BECAUSE THE TAILS MUST NOT DRIFT. The two arms phrase their
|
||||
heads differently ("may apply here" vs "may apply to this Bash call") and
|
||||
that difference is deliberate. Everything after it must not differ, and
|
||||
#3497's history is that the pre-tool arm inherited a defect from its
|
||||
sibling by being modelled on it rather than sharing with it. Two copies of
|
||||
a two-branch string is how one branch gets fixed and the other does not.
|
||||
|
||||
WHY A REPEAT GETS A LINE AT ALL. Both arms used to drop a hit whose id was
|
||||
already on the session's exclusion ledger and emit nothing. That is correct
|
||||
only while the session still HOLDS what it was told, and a compaction
|
||||
breaks exactly that: the earlier injection is summarized away while the id
|
||||
stays on the ledger, so the rule is absent from context AND unreachable for
|
||||
the rest of the session (#3749 closes the compaction half; this closes the
|
||||
ordinary half, where a session simply stops holding a line it read an hour
|
||||
ago).
|
||||
|
||||
Only ONE CLAUSE of the original line is false on a repeat — the claim that
|
||||
the rule is not in the session's loaded set. So only that clause changes.
|
||||
Title, trigger and pull pointer are identical either way, the statement is
|
||||
never injected either way, and a repeat therefore costs the same ~40 tokens
|
||||
as a first surfacing and no more.
|
||||
|
||||
DELIBERATELY NOT ASKING THE SESSION WHETHER IT HOLDS THE RULE. A model
|
||||
asked "do you still hold rule 156?" will say yes, and the claim is
|
||||
unverifiable self-report about its own context. The answer is also not
|
||||
needed: the line is cheap enough to always emit and carries its own remedy
|
||||
in both branches. Removing the question removes the fragility rather than
|
||||
managing it.
|
||||
"""
|
||||
trigger = (rule.when_to_apply or "").strip()
|
||||
tail = (
|
||||
f"You saw it earlier this session; pull it with get_rule({rule.id}) "
|
||||
"if you no longer hold it."
|
||||
if seen else
|
||||
f"Read it with get_rule({rule.id}) before deciding it does not "
|
||||
"apply; it is not in this session's loaded set."
|
||||
)
|
||||
return (
|
||||
f"Standing rule that may apply {where} \u2014 \u201c{rule.title}\u201d"
|
||||
+ (f" ({trigger})" if trigger else "")
|
||||
+ f". {tail}"
|
||||
)
|
||||
|
||||
|
||||
async def build_write_path_hint(
|
||||
user_id: int,
|
||||
@@ -1287,9 +1364,13 @@ async def build_write_path_hint(
|
||||
# resembles what is being written, noticed at the moment it is relevant
|
||||
# rather than by being resident in every session.
|
||||
#
|
||||
# CONDITIONAL ONLY. An always-on rule is already in the session; repeating
|
||||
# it here would be noise, and noise on a hint that fires on every write is
|
||||
# how a hint gets ignored.
|
||||
# EVERY TIER, since #3702 — see the note at RULEHINT_LIMIT. This comment
|
||||
# used to read "CONDITIONAL ONLY: an always-on rule is already in the
|
||||
# session, so repeating it here would be noise". That conflated being
|
||||
# PRESENT in context with being SALIENT at the moment the action is taken,
|
||||
# and it is the same conflation #3750 corrects one layer up: a rule the
|
||||
# session was told about an hour ago is not a rule in front of the reader
|
||||
# now.
|
||||
#
|
||||
# Fails open like every other arm: a rule hint must never break a write.
|
||||
rule_ids: list[int] = []
|
||||
@@ -1308,15 +1389,18 @@ async def build_write_path_hint(
|
||||
)
|
||||
rule_ms = (time.perf_counter() - rule_t0) * 1000.0
|
||||
fresh = [(score, rule) for score, rule in hits if rule.id not in already]
|
||||
for _score, rule in fresh:
|
||||
trigger = (rule.when_to_apply or "").strip()
|
||||
# EVERY hit gets a line; `already` only changes the tail (#3750).
|
||||
for _score, rule in hits:
|
||||
lines.append(
|
||||
f"Standing rule that may apply here — \u201c{rule.title}\u201d"
|
||||
+ (f" ({trigger})" if trigger else "")
|
||||
+ f". Read it with get_rule({rule.id}) before deciding it "
|
||||
"does not apply; it is not in this session's loaded set."
|
||||
_rule_hint_line(rule, where="here", seen=rule.id in already)
|
||||
)
|
||||
rule_ids.append(rule.id)
|
||||
# `rule_ids` stays FRESH-ONLY, and that is the whole telemetry story of
|
||||
# this change (#3752). It is what the hook writes to the exclusion
|
||||
# ledger and what `record_rule_surfaced` counts; a referenced rule is
|
||||
# already on the ledger by definition, and counting it as a surfacing
|
||||
# would inflate pull_through's denominator with a choice this arm never
|
||||
# made. A reference is a RENDERING decision, not a retrieval outcome.
|
||||
rule_ids.extend(rule.id for _score, rule in fresh)
|
||||
# TWO tables, and the split is not arbitrary. retrieval_logs is one
|
||||
# row per CALL, keyed on the score distribution a threshold is tuned
|
||||
# from. rule_usage_events is one row per RULE per event, which is the
|
||||
@@ -1412,10 +1496,10 @@ async def build_tool_rule_hint(
|
||||
which tools it watches, so widening the matcher is a `hooks.json` edit with
|
||||
no change here.
|
||||
|
||||
CONDITIONAL ONLY, exactly as the write-path arm — an always-on rule is
|
||||
already resident and repeating it is noise. That filter is also the
|
||||
transition this arm exists to enable: re-tier a rule to `conditional` and
|
||||
it starts arriving here instead of in every session's preamble.
|
||||
EVERY TIER, since #3702 — the tier filter this docstring used to describe
|
||||
is gone from both arms, for the reason recorded at RULEHINT_LIMIT: present
|
||||
in context and salient at the moment are different properties, and only the
|
||||
second is what this arm is for.
|
||||
|
||||
Fails open and returns an empty context on any error: a recall aid may
|
||||
never break the operator's action.
|
||||
@@ -1469,28 +1553,34 @@ async def build_tool_rule_hint(
|
||||
# and the threshold looks wrong when nothing about it is.
|
||||
suppressed=len(hits) - len(fresh),
|
||||
)
|
||||
if not fresh:
|
||||
# `hits`, not `fresh` (#3750). A call whose only hit is a repeat still
|
||||
# has something to say — the arm just says it differently.
|
||||
if not hits:
|
||||
return out
|
||||
|
||||
lines: list[str] = []
|
||||
rule_ids: list[int] = []
|
||||
for _score, rule in fresh:
|
||||
trigger = (rule.when_to_apply or "").strip()
|
||||
lines.append(
|
||||
f"Standing rule that may apply to this {tool_name} call — "
|
||||
f"“{rule.title}”"
|
||||
+ (f" ({trigger})" if trigger else "")
|
||||
+ f". Read it with get_rule({rule.id}) before deciding it "
|
||||
"does not apply; it is not in this session's loaded set."
|
||||
lines = [
|
||||
_rule_hint_line(
|
||||
rule, where=f"to this {tool_name} call",
|
||||
seen=rule.id in already,
|
||||
)
|
||||
rule_ids.append(rule.id)
|
||||
for _score, rule in hits
|
||||
]
|
||||
# FRESH-ONLY, for the reason given on the sibling arm: a reference is a
|
||||
# rendering decision, not a retrieval outcome, and counting it here
|
||||
# would inflate the denominator pull_through is read from.
|
||||
rule_ids = [rule.id for _score, rule in fresh]
|
||||
|
||||
# RANKED, not ambient: this arm chose what it showed, so a pull can
|
||||
# settle whether the choice was any good. `rule_usage.RANKED_SOURCES`
|
||||
# carries the same name.
|
||||
record_rule_surfaced(
|
||||
user_id=user_id, rule_ids=rule_ids, source="pre_tool_rule",
|
||||
)
|
||||
#
|
||||
# GUARDED, which it did not need to be before #3750: `fresh` can now be
|
||||
# empty on a call that still emitted a line, and recording a surfacing
|
||||
# of nothing would write an event with no rules in it.
|
||||
if rule_ids:
|
||||
record_rule_surfaced(
|
||||
user_id=user_id, rule_ids=rule_ids, source="pre_tool_rule",
|
||||
)
|
||||
out["context"] = "\n".join(lines)
|
||||
out["rule_ids"] = rule_ids
|
||||
except Exception:
|
||||
|
||||
Reference in New Issue
Block a user