docs(6209): rule usage, write path and telemetry tests state the behaviour they pin
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / integration (push) Successful in 1m18s
CI & Build / Python tests (push) Successful in 2m2s
CI & Build / Build & push image (push) Successful in 17s

Docstrings, comments and assertion messages in the three most-cited test
files now say which property each test holds and why, without the issue,
milestone or rule numbers, dates and one-install measurements that recorded
how each property was discovered. Fixture ids and calibration scores stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:18:47 -04:00
co-authored by Claude Opus 5.5
parent bdd803537b
commit 8518afb54d
3 changed files with 290 additions and 351 deletions
+115 -143
View File
@@ -1,9 +1,7 @@
"""Both ends of the rule-usage loop are actually wired (milestone 333 step 2).
"""Both ends of the rule-usage loop are actually wired.
Step 1 built the table and the service. A counter nobody calls reads zero and
looks exactly like a surface nobody uses — which is #2663's shape and the whole
reason this milestone exists. So this file is about the CALL SITES, not the
storage.
A counter nobody calls reads zero and looks exactly like a surface nobody uses.
So this file is about the CALL SITES, not the storage.
Cross-cutting on purpose: the surfaced end lives in `plugin_context`, the pull
end in two different doors, and the property under test is that they meet. Split
@@ -39,8 +37,8 @@ pytestmark = pytest.mark.usefixtures("_bind_user")
# staleness, no synced record, no prior-art menu, no shape signal. The rule arm
# sits deliberately on the FAR side of that guard, because it runs a semantic
# search and moving it above would mean an embedding query on every write in
# the session (#3311's closing note, and the reason its gating is a separate
# question from precision).
# the session (which is why its gating is a separate question from
# precision).
#
# So a fixture that stubs every other arm to empty never reaches the rule arm
# at all — which is what the first run of this file did. The note hit below is
@@ -60,7 +58,7 @@ def _arm_patches(pc, hits, recorder, prior_art=None, cfg=None, rule_search=None,
"""
return (
patch.object(pc, "get_writepath_config",
# Every key, derived from the surface registry (#4102).
# Every key, derived from the surface registry.
# A stub missing one does not fail where a reader would
# see it: the arm fails open, so the KeyError becomes an
# empty hint and every case in _ARMS reports the arm went
@@ -145,7 +143,7 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
"""The consuming half of step 4. `get_writepath_config` assembling a
separate `rule_threshold` means nothing if the arm still passes
`cfg["threshold"]` to its search — the split would exist in the config and
not in the behaviour, and #3311 would be exactly where it was.
not in the behaviour.
The two values are deliberately different here so the assertion can tell
them apart.
@@ -166,12 +164,8 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
kw = search.await_args.kwargs
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
assert kw["limit"] == pc.RULEHINT_LIMIT
# NO CATEGORY FILTER (#3702, repointed by 394). This asserted that the arm
# passed no `tier`. That parameter no longer exists, so the assertion had
# become one that could not fail — which rule 167 rates below having none.
#
# The claim it was making is still live, on the axis that DOES still exist:
# the act arms narrow by nothing, so a preference reaches a write exactly
# NO CATEGORY FILTER, on the axis that exists: the act arms narrow by
# nothing, so a preference reaches a write exactly
# as a rule does. "Already in the session" was never the same as "in front
# of the reader at the moment it applies", and relevance is the
# threshold's job rather than a category's. The one place a kind filter
@@ -191,9 +185,9 @@ async def test_the_arm_does_not_fire_on_a_write_that_matched_nothing():
A write matching no prior art returns before the rule arm runs. That is
deliberate: the arm is a semantic search, and ungating it means an
embedding query on every write in the session. #3311 is explicit that the
gate stays until the arm's precision is fixed, so this failing is a signal
to go read that issue rather than to update the assertion.
embedding query on every write in the session. The gate stays until the
arm's precision is fixed, so this failing is a reason to look at precision
rather than to update the assertion.
"""
rec = MagicMock()
hits = [(0.71, fake_rule(id=156, title="A wait with no deadline is a bug"))]
@@ -300,11 +294,10 @@ def test_the_bulk_loaders_are_not_counted_as_pulls():
)
# ── The AMBIENT end: bulk deliveries (#3473) ───────────────────────────
# ── The AMBIENT end: bulk deliveries ──────────────────────────────────
#
# The preload was the largest rule surface in the product and emitted nothing,
# so its cost was certain and its usefulness unfalsifiable. These assert the
# three delivery shapes now emit — and, just as importantly, that the two
# A bulk delivery that emitted nothing would have a certain cost and an
# unfalsifiable usefulness. These assert the three delivery shapes emit — and, just as importantly, that the two
# lookalike call sites which show nobody anything do NOT.
@@ -315,10 +308,9 @@ def test_the_bulk_loaders_are_not_counted_as_pulls():
def test_rules_payload_records_both_the_family_and_project_halves():
"""One emit site for all five `rules_payload` surfaces.
Per-caller emission would be five sites to remember, and #3430 gap 2 is
what that costs: the process→skill sync went un-emitted through an entire
dedicated telemetry survey because nothing forced its surface to be
accounted for.
Per-caller emission would be five sites to remember, and a site nobody
remembers goes un-emitted with nothing forcing its surface to be accounted
for.
"""
from scribe.services import rulebooks as svc
@@ -341,9 +333,8 @@ def test_rules_payload_records_both_the_family_and_project_halves():
def test_brief_rules_payload_lists_titles_and_records_only_what_it_shows():
"""The handshake's form (#4045): project rules by id and title, nothing
else (subscriptions went in milestone 414). A global rule it doesn't show
must not count as surfaced."""
"""The handshake's form: project rules by id and title, nothing else. A
global rule it doesn't show must not count as surfaced."""
from scribe.services import rulebooks as svc
rec = MagicMock()
@@ -381,17 +372,16 @@ def test_every_rules_payload_caller_names_itself():
# ── The PRE-TOOL arm: rules keyed on the action (#3476) ────────────────
# ── The PRE-TOOL arm: rules keyed on the action ───────────────────────
#
# The write-path arm can only be reached by a code write, so every rule about
# which tool to reach for was unretrievable at the moment it mattered — which
# is why they all had to be resident. These cover the surface that changes it.
# The write-path arm can only be reached by a code write, so a rule about
# which tool to reach for needs an arm keyed on the command. These cover it.
def _tool_patches(pc, hits, recorder, cfg=None, retrieval_log=None):
return (
patch.object(pc, "get_writepath_config",
# Every key, derived from the surface registry (#4102).
# Every key, derived from the surface registry.
# A stub missing one does not fail where a reader would
# see it: the arm fails open, so the KeyError becomes an
# empty hint and every case in _ARMS reports the arm went
@@ -415,13 +405,12 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
return await pc.build_tool_rule_hint(1, tool, command, **kwargs)
# ── the prompt-boundary arm (#3852) ─────────────────────────────────────
# ── the prompt-boundary arm ─────────────────────────────────────────────
#
# The third arm, and it joins _ARMS rather than getting a test file of its
# own. That is the point of the shared parametrisation: #3497's history is
# that the pre-tool arm inherited a defect from its sibling by being MODELLED
# on it instead of sharing with it, and a third arm modelled on two is two
# chances to repeat that. Everything in the family — repeat rendering,
# own. That is the point of the shared parametrisation: an arm MODELLED on
# its sibling instead of sharing with it inherits the sibling's defects, and a
# third arm modelled on two is two chances to repeat one. Everything in the family — repeat rendering,
# fresh-only counting, the log-before-bailout order, the kind register, the
# two recorders reading one list — is a property of every arm or of none.
#
@@ -432,7 +421,7 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
def _prompt_patches(pc, hits, recorder, retrieval_log=None):
return (
# The arm reads its own floor and budget from the surface registry
# rather than a shared config object (#4102) — a third corpus, with a
# rather than a shared config object — a third corpus, with a
# pair nothing has yet tuned for it.
patch.object(rs, "get_setting", AsyncMock(return_value="0.6")),
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
@@ -509,7 +498,7 @@ async def test_the_prompt_arm_says_nothing_when_asked_nothing():
out = await pc.build_prompt_rule_hint(1, " ")
# NO `checkpoint` KEY, and that is the distinction rather than an
# oversight (#4214). The two ACT arms can hold a call because there is
# oversight. The two ACT arms can hold a call because there is
# a composed act to hold; this arm fires on the operator's message,
# before anything has been decided, so there is nothing to put a rule
# in front of. A checkpoint here would have to guess at an act.
@@ -520,7 +509,7 @@ async def test_the_prompt_arm_says_nothing_when_asked_nothing():
@pytest.mark.asyncio
async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
"""#3497's defect, pinned on the arm that most needs it.
"""The log-before-bailout order, pinned on the arm that most needs it.
This bar is INHERITED from the act arms and unverified against prose. The
zero rows are therefore the whole evidence base for whether 0.72 belongs
@@ -531,10 +520,10 @@ async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
out = await _run_prompt_arm([], MagicMock(), retrieval_log=log)
assert out["context"] == ""
# BY SOURCE, not by count. The reserved slot (#3894) logs its own query on
# the same call, so a bare call_count would pin the number of arms rather
# than the property — and would go red the next time one is added, which
# is rule 167's false alarm about the very thing being protected.
# BY SOURCE, not by count. The reserved slot logs its own query on the
# same call, so a bare call_count would pin the number of arms rather than
# the property — and would go red the next time one is added, a false
# alarm about the very thing being protected.
general = [c for c in log.call_args_list
if c.kwargs.get("source") == "prompt_rule"]
assert len(general) == 1, (
@@ -582,8 +571,8 @@ async def test_turning_off_the_notes_menu_does_not_turn_off_rules():
@pytest.mark.asyncio
async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
"""The 2026-09-03 incident in one test: reaching for curl against the forge
API is a Bash call, and nothing watched Bash."""
"""Reaching for curl against the forge API is a Bash call, and this arm is
what watches Bash."""
rec = MagicMock()
hits = [(0.71, fake_rule(id=161,
title="Reach the forge through its MCP tools, never curl",
@@ -601,8 +590,7 @@ async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
async def test_the_tool_arm_is_a_ranked_source():
"""It CHOSE what it showed, so a pull can settle whether the choice was any
good — unlike a preload, which chose nothing. If this drifts into the
ambient class the arm becomes unjudgeable, which is the state #3311
described and M333 existed to end."""
ambient class the arm becomes unjudgeable."""
from scribe.services.rule_usage import is_ambient
assert not is_ambient("pre_tool_rule")
@@ -612,9 +600,8 @@ async def test_the_tool_arm_is_a_ranked_source():
async def test_a_rule_the_session_already_holds_is_referenced_not_re_offered():
"""WAS `..._is_not_re_offered`, asserting `"161" not in out["context"]`.
That assertion was the old contract and #3750 deliberately reverses half of
it: a rule already on the ledger now gets a line with a different tail
instead of being dropped in silence. Withholding was only ever right while
Half of that holds and half is reversed: a rule already on the ledger gets
a line with a different tail instead of being dropped in silence. Withholding was only ever right while
the session still HELD what it was told, and a compaction breaks exactly
that while leaving the id excluded.
@@ -633,7 +620,7 @@ async def test_a_rule_the_session_already_holds_is_referenced_not_re_offered():
"actually chosen by this arm"
)
assert "get_rule(161)" in out["context"], (
"the held rule vanished instead of being referenced (#3750)"
"the held rule vanished instead of being referenced"
)
assert "get_rule(12)" in out["context"], (
"the fresh rule was lost while adding the reference — both belong in "
@@ -729,7 +716,7 @@ def test_the_tool_arm_is_registered_on_bash():
def test_the_hook_and_the_route_agree_on_every_parameter_name():
"""Rule 33, on a brand-new integration between layers.
"""Two layers that only this test connects agree on their contract.
The hook is shell and the route is Python; nothing but this test connects
them. A renamed query arg fails SILENTLY — the route reads an absent value,
@@ -744,16 +731,16 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
handler = route.split("async def pre_tool_rules")[1].split("\n@plugin_bp")[0]
sent = set(re.findall(r"[?&]([a-z_]+)=", hook))
# `root` and `cwd` (milestone 444): where the command runs, so the paths it
# `root` and `cwd`: where the command runs, so the paths it
# names can be made repo-relative for the rulings arm.
assert sent == {"tool", "command", "exclude_rule_ids", "root", "cwd"}, sent
# The project-scope key is the shared helper's to choose (#4085): the hook
# The project-scope key is the shared helper's to choose: the hook
# splices in `scribe_scope_query`'s output, which is `repo=` inside a git
# repo and `project_id=` where a `.scribe` marker names the project. Both
# halves are one contract with the route, so both are pinned.
assert "scribe_scope_query" in hook, "hook no longer asks for a project scope"
# `held_rule_ids` joins them for the same reason (#4100): the OPENED ledger
# `held_rule_ids` joins them for the same reason: the OPENED ledger
# is read by a shared helper so its ageing and format cannot drift from the
# naming ledger's, which means the key is spelled in defs rather than here.
assert "scribe_held_query" in hook, "hook no longer sends the opened ledger"
@@ -765,7 +752,7 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
assert "printf '&held_rule_ids=" in defs, (
"the opened ledger's query key is no longer spelled in the helper"
)
# The rulings ledger, shared with the write-path hook (milestone 444) —
# The rulings ledger, shared with the write-path hook —
# spelled once in its helper for the same reason.
assert "scribe_rulings_query" in hook, "hook no longer sends the rulings ledger"
assert "printf '&seen_ruling_systems=" in defs, (
@@ -790,12 +777,12 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
)
# ── The CALL log is unconditional; the SURFACING log is not (#3497) ────
# ── The CALL log is unconditional; the SURFACING log is not ───────────
#
# Both arms used to write their retrieval_logs row inside a guard on having
# results, so `zero_result_calls` was pinned at 0 and `cleared_threshold` at
# `calls` by the shape of the code — at any threshold whatsoever. #3311 read
# that as a measurement of the corpus and a milestone was scoped on it.
# A retrieval_logs row written inside a guard on having results pins
# `zero_result_calls` at 0 and `cleared_threshold` at `calls` by the shape of
# the code — at any threshold whatsoever — and reads like a measurement of the
# corpus.
#
# The distinction these tests hold: a CALL happened whether or not it found
# anything, and the calls that found nothing are the only evidence a threshold
@@ -842,7 +829,7 @@ async def test_the_tool_arm_logs_the_call_that_found_nothing():
"""It matters more here than on the sibling. This arm fires on every Bash
call, so an empty `sources` row is the normal outcome — and with no row at
all, "the ranker declined" is indistinguishable from "the hook never fired",
which is exactly the silent failure the arm was built to stop (#3476)."""
which is exactly the silent failure the arm was built to stop."""
log, rec = MagicMock(), MagicMock()
out = await _run_tool_arm([], rec, retrieval_log=log)
@@ -890,10 +877,9 @@ async def test_a_command_the_arm_never_searched_writes_no_row_at_all():
def test_no_rule_arm_logs_its_call_behind_a_results_guard():
"""Structural, on top of the behavioural tests above, because the defect
was one level of indentation and it appeared INDEPENDENTLY in two arms —
the pre-tool arm inherited it by being modelled on its sibling (#3497).
the pre-tool arm inherited it by being modelled on its sibling.
Since milestone 456 there is one implementation to guard rather than a
copy per arm, so the property is checked where it is written: the
There is one implementation to guard rather than a copy per arm, so the property is checked where it is written: the
pipeline's ranked-search stage writes the call row unconditionally, the
arm runner calls that stage before it can return, and the surfacing row
alone stays behind `if rule_ids:`. And no arm writes a rule-source row of
@@ -921,7 +907,7 @@ def test_no_rule_arm_logs_its_call_behind_a_results_guard():
assert not early, (
f"the ranked-search stage can return at line {early[0]} before logging "
f"its call — a call that found nothing is the only evidence a bar is "
f"too high (#3497)"
f"too high"
)
# 2. The runner reaches the stage before its first early return.
@@ -958,7 +944,7 @@ def test_no_rule_arm_logs_its_call_behind_a_results_guard():
)
# ── Suppression: which zeros were the ranker, which were repeats (#3497) ──
# ── Suppression: which zeros were the ranker, which were repeats ──────
#
# Making the call log unconditional exposed a second ambiguity in the same row.
# A zero-result rule call is two unrelated events: the ranker found nothing
@@ -1020,19 +1006,15 @@ async def test_a_shown_hit_is_not_counted_as_suppressed():
assert len(log.call_args.kwargs["results"]) == 1
# ── The identity that falsified this milestone (#3668) ─────────────────
# ── The identity between the two tables ─────────────────────────────────
#
# `rule_usage.surfaced` == `pre_tool_rule.cleared` + `write_path_rule.cleared`.
# Milestone #379 was scoped on a reconstruction that put ~64% of ranked rule
# surfacings as never reaching `rule_usage_events`. Five steps were planned
# against it. One read of this identity — 17 = 17, then 39 = 39 on a second
# window — falsified the whole thing: the gap was two counters that started
# recording on different days, not a write path dropping rows.
# Two counters that started recording on different days produce a gap that
# reads like a write path dropping rows; this identity tells the two apart.
#
# So the identity is not a nice-to-have. It is the cheapest true statement
# available about this pair of tables, and its absence is what let a magnitude
# that merely LOOKED wrong survive a code review and a five-step plan. An
# identity that must hold exactly beats a magnitude that looks wrong.
# available about this pair of tables. An identity that must hold exactly
# beats a magnitude that looks wrong.
#
# WHY THE ARM IS THE RIGHT PLACE TO PIN IT, and the readout is not. Inside an
# arm, one `fresh` list feeds both recorders in one function, so the counts
@@ -1040,8 +1022,8 @@ async def test_a_shown_hit_is_not_counted_as_suppressed():
# than it looks: `cleared_threshold` counts CALLS that beat the bar while
# `surfaced` counts RULES, and those coincide only while `RULEHINT_LIMIT` is 1.
# Raise the limit and the readout identity breaks while nothing is wrong.
# `RULEHINT_LIMIT` has already moved once (2 → 1, `2385100`), and that move is
# half of why the original reconstruction misread its own numbers.
# `RULEHINT_LIMIT` can move, and a reader comparing windows across a move
# misreads its own numbers.
#
# Hence three hits below, where production currently returns at most one. The
# test is deliberately in a state the limit does not permit today, because what
@@ -1055,7 +1037,7 @@ _THREE_HITS = [
]
def test_the_three_hit_fixture_sits_inside_the_rule_band():
"""The fixture's own precondition, asserted rather than commented (#3851).
"""The fixture's own precondition, asserted rather than commented.
The act arms band before they dedup, so a fixture whose spread straddles
`_RULEHINT_BAND` loses its lowest hit to the BAND and then reports a count
@@ -1085,8 +1067,7 @@ _ARMS = [
("write_path_rule", _run_arm),
("pre_tool_rule", _run_tool_arm),
# The prompt arm joins the family rather than being modelled on it — see
# the block above _prompt_patches for why that distinction is the whole
# lesson of #3497.
# the block above _prompt_patches for why that distinction matters.
("prompt_rule", _run_prompt_arm),
]
@@ -1140,7 +1121,7 @@ async def test_both_recorders_report_the_same_rules_for_one_call(
f"call log recorded {logged} and the surfacing log recorded {surfaced}. "
f"Both come from `fresh`, in one function, so any difference is a bug "
f"in the wiring — and it is the shape that reads as a lost write when "
f"the two tables are later compared in aggregate (#3668)."
f"the two tables are later compared in aggregate."
)
assert len(logged) == expected, (
"the fixture stopped exercising what it claims to; check the exclusion "
@@ -1148,18 +1129,15 @@ async def test_both_recorders_report_the_same_rules_for_one_call(
)
# ── A repeat is REFERENCED, not withheld (#3750) ──────────────────────────
# ── A repeat is REFERENCED, not withheld ─────────────────────────────────
#
# Both arms used to drop a hit already on the session's exclusion ledger and
# emit nothing at all. That is right only while the session still HOLDS what it
# was told — and it stops being right the moment a compaction summarizes the
# Dropping a hit already on the session's exclusion ledger and emitting
# nothing is right only while the session still HOLDS what it was told — and it stops being right the moment a compaction summarizes the
# earlier injection away while the id stays on the ledger, which leaves the
# rule absent from context AND unreachable for the rest of the session.
#
# The tests below pin the emitted LINE, not the prose describing it, and both
# arms are parametrized through one body so the two tails cannot drift apart —
# #3497's history is that the pre-tool arm inherited a defect from its sibling
# by being modelled on it rather than sharing with it.
# arms are parametrized through one body so the two tails cannot drift apart.
_HELD = fake_rule(
id=156,
@@ -1168,11 +1146,10 @@ _HELD = fake_rule(
when_to_apply="writing any call that crosses a process boundary",
)
_FRESH_TAIL = "not in this session's loaded set"
# `seen` WITHOUT `held` is the middle state since #4100: an arm named this rule
# earlier, and the session never opened it. Every test below drives the arms
# with `exclude_rule_ids` alone — the naming ledger — so this is the tail they
# produce, and it is the one that used to claim "You saw it earlier this
# session" about a teaser nobody had read.
# `seen` WITHOUT `held` is the middle state: an arm named this rule earlier,
# and the session never opened it. Every test below drives the arms with
# `exclude_rule_ids` alone — the naming ledger — so this is the tail they
# produce, and it must not claim the session read a teaser nobody opened.
_SEEN_TAIL = "Mentioned earlier this session but not opened"
# The third state, reached only when the opened ledger names the rule too.
_HELD_TAIL = "You opened it earlier this session"
@@ -1181,19 +1158,18 @@ _HELD_TAIL = "You opened it earlier this session"
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
"""THE REGRESSION, stated as the thing that used to be absent.
"""A repeat still renders a line.
Falsified against the old behaviour: before #3750 both arms filtered to
`fresh` before rendering, so an all-excluded call returned an empty
context and this assertion fails on `context == ""`.
An arm that filters to `fresh` before rendering returns an empty context
for an all-excluded call, and this assertion fails on `context == ""`.
"""
out = await run([(0.81, _HELD)], MagicMock(), exclude_rule_ids=[156])
# ON `get_rule(156)` RATHER THAN A TRUTHY CONTEXT. The write-path arm's
# context also carries the prior-art menu, the staleness line and the shape
# signals, so `assert out["context"]` is TRUE under the old behaviour and
# would pin nothing on that arm while looking identical to a real check on
# the other. The rule line is the only part of the string this changes.
# signals, so `assert out["context"]` is TRUE even when the rule line is
# missing, and would pin nothing on that arm while looking identical to a
# real check on the other. The rule line is the part under test.
assert "get_rule(156)" in out["context"], (
f"{source} emitted no rule line for a rule the session had already "
f"been shown. Silence is only correct while the session still holds "
@@ -1229,14 +1205,14 @@ async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, ru
)
assert fresh_ctx != seen_ctx, "the two tails collapsed into one"
# Neither call named an OPENED ledger, so neither may claim the session
# read anything (#4100). Catches `held` defaulting true, which would make
# read anything. Catches `held` defaulting true, which would make
# every repeat assert the strongest of the three claims on no evidence.
assert _HELD_TAIL not in fresh_ctx and _HELD_TAIL not in seen_ctx, (
f"{source} said the session had opened a rule it was only shown"
)
# ── the third axis: KIND (milestone 399) ────────────────────────────────
# ── the third axis: KIND ─────────────────────────────────────────────────
#
# A preference is a rule row that does not bind, so it rides the same arms and
# the same line. What must differ is the REGISTER: a rule's line tells the
@@ -1245,8 +1221,8 @@ async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, ru
# to find how this has been done, and following it buys consistency rather
# than correctness.
#
# Rendered in the rule's voice, a preference becomes the thing milestone 399
# exists to avoid: a rule with a different column.
# Rendered in the rule's voice, a preference becomes a rule with a different
# column.
_PREF = fake_rule(
id=177,
@@ -1290,7 +1266,7 @@ async def test_kind_and_seen_do_not_read_each_other(source, run):
Whether a record is already on the exclusion ledger has nothing to do with
how much force it carries. Keeping the two unrelated in the code is what
let a second kind arrive without reopening #3750's repeat question — and
let a second kind arrive without reopening the repeat question — and
the way that silently breaks is a `seen` branch that grows a kind test,
or a kind branch that grows a `seen` test, leaving one of the four
combinations rendered by nobody's intention.
@@ -1346,9 +1322,9 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
)
# ── What a reference IS in the telemetry: nothing new (#3752) ─────────────
# ── What a reference IS in the telemetry: nothing new ──────────────────────
#
# THE RELATION, STATED BEFORE IT SHIPS. #3750 changes what is RENDERED and
# THE RELATION. Rendering a repeat as a reference changes what is RENDERED and
# nothing about what is COUNTED:
#
# result_count counts fresh surfacings — unchanged
@@ -1357,13 +1333,12 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
#
# A reference is a rendering decision, not a retrieval outcome. That answer is
# not free: the naive implementation renders repeats by dropping the `fresh`
# filter, which takes `suppressed_count` to zero everywhere — and #3739's
# near-miss fix identifies repeat-caused zeros by `suppressed_count > 0`, so
# the contamination corrected on 2026-09-08 would return by a different route,
# in the same field, with the fix still sitting in the code not working.
# filter, which takes `suppressed_count` to zero everywhere — and the
# near-miss readout identifies repeat-caused zeros by `suppressed_count > 0`,
# so every repeat would be re-counted as a genuine ranker rejection.
#
# #3712 gave a reader `complete_from` for a counter that started late. Nothing
# tells a reader a counter's DEFINITION moved. This test is the cheap version
# `complete_from` tells a reader a counter started late. Nothing tells a
# reader a counter's DEFINITION moved. This test is the cheap version
# of that guarantee: the claim "nothing moved" is only worth anything if it is
# checkable, so it is asserted rather than described.
@@ -1371,7 +1346,7 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
@pytest.mark.asyncio
async def test_a_reference_is_rendered_but_not_counted(source, run):
"""The counters must read exactly as they did before #3750."""
"""Rendering a repeat leaves every counter as it was."""
log, rec = MagicMock(), MagicMock()
out = await run([(0.81, _HELD)], rec, retrieval_log=log,
exclude_rule_ids=[156])
@@ -1384,7 +1359,7 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
f"is not evidence the bar is set correctly."
)
assert row["suppressed"] == 1, (
f"{source} stopped reporting the repeat as suppressed. #3739's "
f"{source} stopped reporting the repeat as suppressed. The "
f"near_misses predicate excludes declines with suppressed_count > 0 — "
f"if this reads 0, every repeat-caused zero is re-counted as a genuine "
f"ranker rejection and the near-miss contamination returns."
@@ -1406,7 +1381,7 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
)
# ── the reserved preference slot (#3894) ────────────────────────────────
# ── the reserved preference slot ────────────────────────────────────────
#
# A rule and a preference are not equally served by one ranking, because their
# losses are not equal. A rule crowded out at the prompt boundary still fires
@@ -1416,9 +1391,9 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
#
# The failure is invisible without this slot. The rule that won is a
# legitimate hit, the telemetry reads healthy, and the only symptom is a
# preference that quietly never arrives — which is `reuse_slot`'s shape one
# corpus over (#2463), where snippets kept losing to project records that
# merely resembled the query.
# preference that quietly never arrives — `reuse_slot`'s shape one corpus
# over, where snippets lose to project records that merely resemble the
# query.
def _search_by_kind(general, preference):
@@ -1537,9 +1512,8 @@ async def test_the_slot_extends_rather_than_displacing():
`reuse_slot` evicts its menu's weakest hit; this one does not, and the
reason is the ledger. A displaced hit sits in `prompt_rule`'s
retrieval_logs row while never being surfaced, so that source's two
tables stop agreeing — and #3668's identity is the cheapest true
statement available about this pair. Milestone #379 is what losing it
costs: five steps planned against two counters disagreeing.
tables stop agreeing — and the identity is the cheapest true statement
available about this pair.
"""
log, rec = MagicMock(), MagicMock()
out, _ = await _run_slot(
@@ -1562,7 +1536,7 @@ async def test_the_slot_extends_rather_than_displacing():
]
assert logged == surfaced, (
f"prompt_rule logged {logged} and surfaced {surfaced} — the identity "
f"#3668 pins, broken by the slot rather than by a write path"
f"pinned above, broken by the slot rather than by a write path"
)
@@ -1570,9 +1544,9 @@ async def test_the_slot_extends_rather_than_displacing():
async def test_the_slot_accounts_for_itself_under_its_own_source():
"""Both sides of the trade, logged.
#2463's own finding is the warning rather than the precedent: the hit
that slot pushed OUT was in retrieval_logs while the query that pushed it
out was not, so the slot could never be judged against what it displaced.
A slot whose query is not logged cannot be judged against what it
displaced: the hit it pushed OUT is in retrieval_logs and the query that
pushed it out is not.
This one logs its query AND records its surfacing, under a source of its
own, so it can be evaluated separately from the ranking it bypassed.
"""
@@ -1597,9 +1571,9 @@ async def test_a_preference_on_the_ledger_keeps_the_slot_and_is_not_recounted():
A preference is the kind of record where being reminded is the point, so
one already on the ledger occupies the slot rather than being skipped for
a fresh one — rendered with the repeat tail (#3750). What it must not do
is register a second surfacing, which would count one delivery twice in
the denominator pull-through is read from (#3752).
a fresh one — rendered with the repeat tail. What it must not do is
register a second surfacing, which would count one delivery twice in the
denominator pull-through is read from.
"""
log, rec = MagicMock(), MagicMock()
out, _ = await _run_slot(
@@ -1616,19 +1590,17 @@ async def test_a_preference_on_the_ledger_keeps_the_slot_and_is_not_recounted():
)
assert 140 not in out["rule_ids"], (
"a repeat was written back to the hook's ledger, which would keep "
"pushing its stamp forward so it never aged out (#3751)"
"pushing its stamp forward so it never aged out"
)
# ── each act arm uses its OWN bar, end to end (#3853) ───────────────────
# ── each act arm uses its OWN bar, end to end ───────────────────────────
#
# The two act arms shared one threshold until the telemetry showed them
# behaving like different subsystems at the same number: write_path_rule
# speaking on 37% of 2,325 calls, pre_tool_rule on 2% of 11,768, because a
# The two act arms behave like different subsystems at the same number: a
# code payload is long and rich where a shell command is short and carries
# less signal for the same relevance.
#
# Splitting the bar creates a failure the old single-bar code could not have:
# Separate bars create a failure a single bar could not have:
# an arm can now search at one threshold and REPORT another. That row is what
# near-miss analysis is read against, so a mismatch does not look like a bug —
# it looks like a corpus whose scores sit somewhere they do not, and it would
@@ -1652,7 +1624,7 @@ async def test_each_act_arm_searches_at_its_own_bar():
assert search.await_args.kwargs["threshold"] == 0.61, (
"the command arm searched at the write-path arm's bar; the two were "
"split at #3853 precisely because one number cannot serve both"
"split precisely because one number cannot serve both"
)
@@ -1685,11 +1657,11 @@ async def test_an_act_arm_reports_the_bar_it_actually_searched_at():
assert rows[0].kwargs["threshold"] == search.await_args.kwargs["threshold"]
# ── scope: a session gets global rules plus its own project's (milestone 414) ──
# ── scope: a session gets global rules plus its own project's ──
def test_every_hook_rule_search_says_which_project_it_is_for():
"""Every call site passes `project_id`, walked rather than grepped (rule 167).
"""Every call site passes `project_id`, walked rather than grepped.
semantic_search_rules defaults to GLOBAL rules only, so an arm that forgets
the keyword does not leak another project's rules — it quietly stops
@@ -1698,7 +1670,7 @@ def test_every_hook_rule_search_says_which_project_it_is_for():
`everywhere` is not an acceptable answer in a hook, which speaks unasked.
"""
sources = {
# Since milestone 456 the hook arms search through the pipeline's one
# The hook arms search through the pipeline's one
# ranked-search stage, checked below — so a direct search appearing
# here again is a copy of the arm coming back, and fails the count.
"src/scribe/services/plugin_context.py": 0,