docs(6209): rule usage, write path and telemetry tests state the behaviour they pin
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / integration (push) Successful in 1m18s
CI & Build / Python tests (push) Successful in 2m2s
CI & Build / Build & push image (push) Successful in 17s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / integration (push) Successful in 1m18s
CI & Build / Python tests (push) Successful in 2m2s
CI & Build / Build & push image (push) Successful in 17s
Docstrings, comments and assertion messages in the three most-cited test files now say which property each test holds and why, without the issue, milestone or rule numbers, dates and one-install measurements that recorded how each property was discovered. Fixture ids and calibration scores stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+115
-143
@@ -1,9 +1,7 @@
|
||||
"""Both ends of the rule-usage loop are actually wired (milestone 333 step 2).
|
||||
"""Both ends of the rule-usage loop are actually wired.
|
||||
|
||||
Step 1 built the table and the service. A counter nobody calls reads zero and
|
||||
looks exactly like a surface nobody uses — which is #2663's shape and the whole
|
||||
reason this milestone exists. So this file is about the CALL SITES, not the
|
||||
storage.
|
||||
A counter nobody calls reads zero and looks exactly like a surface nobody uses.
|
||||
So this file is about the CALL SITES, not the storage.
|
||||
|
||||
Cross-cutting on purpose: the surfaced end lives in `plugin_context`, the pull
|
||||
end in two different doors, and the property under test is that they meet. Split
|
||||
@@ -39,8 +37,8 @@ pytestmark = pytest.mark.usefixtures("_bind_user")
|
||||
# staleness, no synced record, no prior-art menu, no shape signal. The rule arm
|
||||
# sits deliberately on the FAR side of that guard, because it runs a semantic
|
||||
# search and moving it above would mean an embedding query on every write in
|
||||
# the session (#3311's closing note, and the reason its gating is a separate
|
||||
# question from precision).
|
||||
# the session (which is why its gating is a separate question from
|
||||
# precision).
|
||||
#
|
||||
# So a fixture that stubs every other arm to empty never reaches the rule arm
|
||||
# at all — which is what the first run of this file did. The note hit below is
|
||||
@@ -60,7 +58,7 @@ def _arm_patches(pc, hits, recorder, prior_art=None, cfg=None, rule_search=None,
|
||||
"""
|
||||
return (
|
||||
patch.object(pc, "get_writepath_config",
|
||||
# Every key, derived from the surface registry (#4102).
|
||||
# Every key, derived from the surface registry.
|
||||
# A stub missing one does not fail where a reader would
|
||||
# see it: the arm fails open, so the KeyError becomes an
|
||||
# empty hint and every case in _ARMS reports the arm went
|
||||
@@ -145,7 +143,7 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
|
||||
"""The consuming half of step 4. `get_writepath_config` assembling a
|
||||
separate `rule_threshold` means nothing if the arm still passes
|
||||
`cfg["threshold"]` to its search — the split would exist in the config and
|
||||
not in the behaviour, and #3311 would be exactly where it was.
|
||||
not in the behaviour.
|
||||
|
||||
The two values are deliberately different here so the assertion can tell
|
||||
them apart.
|
||||
@@ -166,12 +164,8 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
|
||||
kw = search.await_args.kwargs
|
||||
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
|
||||
assert kw["limit"] == pc.RULEHINT_LIMIT
|
||||
# NO CATEGORY FILTER (#3702, repointed by 394). This asserted that the arm
|
||||
# passed no `tier`. That parameter no longer exists, so the assertion had
|
||||
# become one that could not fail — which rule 167 rates below having none.
|
||||
#
|
||||
# The claim it was making is still live, on the axis that DOES still exist:
|
||||
# the act arms narrow by nothing, so a preference reaches a write exactly
|
||||
# NO CATEGORY FILTER, on the axis that exists: the act arms narrow by
|
||||
# nothing, so a preference reaches a write exactly
|
||||
# as a rule does. "Already in the session" was never the same as "in front
|
||||
# of the reader at the moment it applies", and relevance is the
|
||||
# threshold's job rather than a category's. The one place a kind filter
|
||||
@@ -191,9 +185,9 @@ async def test_the_arm_does_not_fire_on_a_write_that_matched_nothing():
|
||||
|
||||
A write matching no prior art returns before the rule arm runs. That is
|
||||
deliberate: the arm is a semantic search, and ungating it means an
|
||||
embedding query on every write in the session. #3311 is explicit that the
|
||||
gate stays until the arm's precision is fixed, so this failing is a signal
|
||||
to go read that issue rather than to update the assertion.
|
||||
embedding query on every write in the session. The gate stays until the
|
||||
arm's precision is fixed, so this failing is a reason to look at precision
|
||||
rather than to update the assertion.
|
||||
"""
|
||||
rec = MagicMock()
|
||||
hits = [(0.71, fake_rule(id=156, title="A wait with no deadline is a bug"))]
|
||||
@@ -300,11 +294,10 @@ def test_the_bulk_loaders_are_not_counted_as_pulls():
|
||||
)
|
||||
|
||||
|
||||
# ── The AMBIENT end: bulk deliveries (#3473) ───────────────────────────
|
||||
# ── The AMBIENT end: bulk deliveries ──────────────────────────────────
|
||||
#
|
||||
# The preload was the largest rule surface in the product and emitted nothing,
|
||||
# so its cost was certain and its usefulness unfalsifiable. These assert the
|
||||
# three delivery shapes now emit — and, just as importantly, that the two
|
||||
# A bulk delivery that emitted nothing would have a certain cost and an
|
||||
# unfalsifiable usefulness. These assert the three delivery shapes emit — and, just as importantly, that the two
|
||||
# lookalike call sites which show nobody anything do NOT.
|
||||
|
||||
|
||||
@@ -315,10 +308,9 @@ def test_the_bulk_loaders_are_not_counted_as_pulls():
|
||||
def test_rules_payload_records_both_the_family_and_project_halves():
|
||||
"""One emit site for all five `rules_payload` surfaces.
|
||||
|
||||
Per-caller emission would be five sites to remember, and #3430 gap 2 is
|
||||
what that costs: the process→skill sync went un-emitted through an entire
|
||||
dedicated telemetry survey because nothing forced its surface to be
|
||||
accounted for.
|
||||
Per-caller emission would be five sites to remember, and a site nobody
|
||||
remembers goes un-emitted with nothing forcing its surface to be accounted
|
||||
for.
|
||||
"""
|
||||
from scribe.services import rulebooks as svc
|
||||
|
||||
@@ -341,9 +333,8 @@ def test_rules_payload_records_both_the_family_and_project_halves():
|
||||
|
||||
|
||||
def test_brief_rules_payload_lists_titles_and_records_only_what_it_shows():
|
||||
"""The handshake's form (#4045): project rules by id and title, nothing
|
||||
else (subscriptions went in milestone 414). A global rule it doesn't show
|
||||
must not count as surfaced."""
|
||||
"""The handshake's form: project rules by id and title, nothing else. A
|
||||
global rule it doesn't show must not count as surfaced."""
|
||||
from scribe.services import rulebooks as svc
|
||||
|
||||
rec = MagicMock()
|
||||
@@ -381,17 +372,16 @@ def test_every_rules_payload_caller_names_itself():
|
||||
|
||||
|
||||
|
||||
# ── The PRE-TOOL arm: rules keyed on the action (#3476) ────────────────
|
||||
# ── The PRE-TOOL arm: rules keyed on the action ───────────────────────
|
||||
#
|
||||
# The write-path arm can only be reached by a code write, so every rule about
|
||||
# which tool to reach for was unretrievable at the moment it mattered — which
|
||||
# is why they all had to be resident. These cover the surface that changes it.
|
||||
# The write-path arm can only be reached by a code write, so a rule about
|
||||
# which tool to reach for needs an arm keyed on the command. These cover it.
|
||||
|
||||
|
||||
def _tool_patches(pc, hits, recorder, cfg=None, retrieval_log=None):
|
||||
return (
|
||||
patch.object(pc, "get_writepath_config",
|
||||
# Every key, derived from the surface registry (#4102).
|
||||
# Every key, derived from the surface registry.
|
||||
# A stub missing one does not fail where a reader would
|
||||
# see it: the arm fails open, so the KeyError becomes an
|
||||
# empty hint and every case in _ARMS reports the arm went
|
||||
@@ -415,13 +405,12 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
|
||||
return await pc.build_tool_rule_hint(1, tool, command, **kwargs)
|
||||
|
||||
|
||||
# ── the prompt-boundary arm (#3852) ─────────────────────────────────────
|
||||
# ── the prompt-boundary arm ─────────────────────────────────────────────
|
||||
#
|
||||
# The third arm, and it joins _ARMS rather than getting a test file of its
|
||||
# own. That is the point of the shared parametrisation: #3497's history is
|
||||
# that the pre-tool arm inherited a defect from its sibling by being MODELLED
|
||||
# on it instead of sharing with it, and a third arm modelled on two is two
|
||||
# chances to repeat that. Everything in the family — repeat rendering,
|
||||
# own. That is the point of the shared parametrisation: an arm MODELLED on
|
||||
# its sibling instead of sharing with it inherits the sibling's defects, and a
|
||||
# third arm modelled on two is two chances to repeat one. Everything in the family — repeat rendering,
|
||||
# fresh-only counting, the log-before-bailout order, the kind register, the
|
||||
# two recorders reading one list — is a property of every arm or of none.
|
||||
#
|
||||
@@ -432,7 +421,7 @@ async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api
|
||||
def _prompt_patches(pc, hits, recorder, retrieval_log=None):
|
||||
return (
|
||||
# The arm reads its own floor and budget from the surface registry
|
||||
# rather than a shared config object (#4102) — a third corpus, with a
|
||||
# rather than a shared config object — a third corpus, with a
|
||||
# pair nothing has yet tuned for it.
|
||||
patch.object(rs, "get_setting", AsyncMock(return_value="0.6")),
|
||||
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
|
||||
@@ -509,7 +498,7 @@ async def test_the_prompt_arm_says_nothing_when_asked_nothing():
|
||||
out = await pc.build_prompt_rule_hint(1, " ")
|
||||
|
||||
# NO `checkpoint` KEY, and that is the distinction rather than an
|
||||
# oversight (#4214). The two ACT arms can hold a call because there is
|
||||
# oversight. The two ACT arms can hold a call because there is
|
||||
# a composed act to hold; this arm fires on the operator's message,
|
||||
# before anything has been decided, so there is nothing to put a rule
|
||||
# in front of. A checkpoint here would have to guess at an act.
|
||||
@@ -520,7 +509,7 @@ async def test_the_prompt_arm_says_nothing_when_asked_nothing():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
|
||||
"""#3497's defect, pinned on the arm that most needs it.
|
||||
"""The log-before-bailout order, pinned on the arm that most needs it.
|
||||
|
||||
This bar is INHERITED from the act arms and unverified against prose. The
|
||||
zero rows are therefore the whole evidence base for whether 0.72 belongs
|
||||
@@ -531,10 +520,10 @@ async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
|
||||
out = await _run_prompt_arm([], MagicMock(), retrieval_log=log)
|
||||
|
||||
assert out["context"] == ""
|
||||
# BY SOURCE, not by count. The reserved slot (#3894) logs its own query on
|
||||
# the same call, so a bare call_count would pin the number of arms rather
|
||||
# than the property — and would go red the next time one is added, which
|
||||
# is rule 167's false alarm about the very thing being protected.
|
||||
# BY SOURCE, not by count. The reserved slot logs its own query on the
|
||||
# same call, so a bare call_count would pin the number of arms rather than
|
||||
# the property — and would go red the next time one is added, a false
|
||||
# alarm about the very thing being protected.
|
||||
general = [c for c in log.call_args_list
|
||||
if c.kwargs.get("source") == "prompt_rule"]
|
||||
assert len(general) == 1, (
|
||||
@@ -582,8 +571,8 @@ async def test_turning_off_the_notes_menu_does_not_turn_off_rules():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
|
||||
"""The 2026-09-03 incident in one test: reaching for curl against the forge
|
||||
API is a Bash call, and nothing watched Bash."""
|
||||
"""Reaching for curl against the forge API is a Bash call, and this arm is
|
||||
what watches Bash."""
|
||||
rec = MagicMock()
|
||||
hits = [(0.71, fake_rule(id=161,
|
||||
title="Reach the forge through its MCP tools, never curl",
|
||||
@@ -601,8 +590,7 @@ async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
|
||||
async def test_the_tool_arm_is_a_ranked_source():
|
||||
"""It CHOSE what it showed, so a pull can settle whether the choice was any
|
||||
good — unlike a preload, which chose nothing. If this drifts into the
|
||||
ambient class the arm becomes unjudgeable, which is the state #3311
|
||||
described and M333 existed to end."""
|
||||
ambient class the arm becomes unjudgeable."""
|
||||
from scribe.services.rule_usage import is_ambient
|
||||
|
||||
assert not is_ambient("pre_tool_rule")
|
||||
@@ -612,9 +600,8 @@ async def test_the_tool_arm_is_a_ranked_source():
|
||||
async def test_a_rule_the_session_already_holds_is_referenced_not_re_offered():
|
||||
"""WAS `..._is_not_re_offered`, asserting `"161" not in out["context"]`.
|
||||
|
||||
That assertion was the old contract and #3750 deliberately reverses half of
|
||||
it: a rule already on the ledger now gets a line with a different tail
|
||||
instead of being dropped in silence. Withholding was only ever right while
|
||||
Half of that holds and half is reversed: a rule already on the ledger gets
|
||||
a line with a different tail instead of being dropped in silence. Withholding was only ever right while
|
||||
the session still HELD what it was told, and a compaction breaks exactly
|
||||
that while leaving the id excluded.
|
||||
|
||||
@@ -633,7 +620,7 @@ async def test_a_rule_the_session_already_holds_is_referenced_not_re_offered():
|
||||
"actually chosen by this arm"
|
||||
)
|
||||
assert "get_rule(161)" in out["context"], (
|
||||
"the held rule vanished instead of being referenced (#3750)"
|
||||
"the held rule vanished instead of being referenced"
|
||||
)
|
||||
assert "get_rule(12)" in out["context"], (
|
||||
"the fresh rule was lost while adding the reference — both belong in "
|
||||
@@ -729,7 +716,7 @@ def test_the_tool_arm_is_registered_on_bash():
|
||||
|
||||
|
||||
def test_the_hook_and_the_route_agree_on_every_parameter_name():
|
||||
"""Rule 33, on a brand-new integration between layers.
|
||||
"""Two layers that only this test connects agree on their contract.
|
||||
|
||||
The hook is shell and the route is Python; nothing but this test connects
|
||||
them. A renamed query arg fails SILENTLY — the route reads an absent value,
|
||||
@@ -744,16 +731,16 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
|
||||
handler = route.split("async def pre_tool_rules")[1].split("\n@plugin_bp")[0]
|
||||
|
||||
sent = set(re.findall(r"[?&]([a-z_]+)=", hook))
|
||||
# `root` and `cwd` (milestone 444): where the command runs, so the paths it
|
||||
# `root` and `cwd`: where the command runs, so the paths it
|
||||
# names can be made repo-relative for the rulings arm.
|
||||
assert sent == {"tool", "command", "exclude_rule_ids", "root", "cwd"}, sent
|
||||
|
||||
# The project-scope key is the shared helper's to choose (#4085): the hook
|
||||
# The project-scope key is the shared helper's to choose: the hook
|
||||
# splices in `scribe_scope_query`'s output, which is `repo=` inside a git
|
||||
# repo and `project_id=` where a `.scribe` marker names the project. Both
|
||||
# halves are one contract with the route, so both are pinned.
|
||||
assert "scribe_scope_query" in hook, "hook no longer asks for a project scope"
|
||||
# `held_rule_ids` joins them for the same reason (#4100): the OPENED ledger
|
||||
# `held_rule_ids` joins them for the same reason: the OPENED ledger
|
||||
# is read by a shared helper so its ageing and format cannot drift from the
|
||||
# naming ledger's, which means the key is spelled in defs rather than here.
|
||||
assert "scribe_held_query" in hook, "hook no longer sends the opened ledger"
|
||||
@@ -765,7 +752,7 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
|
||||
assert "printf '&held_rule_ids=" in defs, (
|
||||
"the opened ledger's query key is no longer spelled in the helper"
|
||||
)
|
||||
# The rulings ledger, shared with the write-path hook (milestone 444) —
|
||||
# The rulings ledger, shared with the write-path hook —
|
||||
# spelled once in its helper for the same reason.
|
||||
assert "scribe_rulings_query" in hook, "hook no longer sends the rulings ledger"
|
||||
assert "printf '&seen_ruling_systems=" in defs, (
|
||||
@@ -790,12 +777,12 @@ def test_the_hook_and_the_route_agree_on_every_parameter_name():
|
||||
)
|
||||
|
||||
|
||||
# ── The CALL log is unconditional; the SURFACING log is not (#3497) ────
|
||||
# ── The CALL log is unconditional; the SURFACING log is not ───────────
|
||||
#
|
||||
# Both arms used to write their retrieval_logs row inside a guard on having
|
||||
# results, so `zero_result_calls` was pinned at 0 and `cleared_threshold` at
|
||||
# `calls` by the shape of the code — at any threshold whatsoever. #3311 read
|
||||
# that as a measurement of the corpus and a milestone was scoped on it.
|
||||
# A retrieval_logs row written inside a guard on having results pins
|
||||
# `zero_result_calls` at 0 and `cleared_threshold` at `calls` by the shape of
|
||||
# the code — at any threshold whatsoever — and reads like a measurement of the
|
||||
# corpus.
|
||||
#
|
||||
# The distinction these tests hold: a CALL happened whether or not it found
|
||||
# anything, and the calls that found nothing are the only evidence a threshold
|
||||
@@ -842,7 +829,7 @@ async def test_the_tool_arm_logs_the_call_that_found_nothing():
|
||||
"""It matters more here than on the sibling. This arm fires on every Bash
|
||||
call, so an empty `sources` row is the normal outcome — and with no row at
|
||||
all, "the ranker declined" is indistinguishable from "the hook never fired",
|
||||
which is exactly the silent failure the arm was built to stop (#3476)."""
|
||||
which is exactly the silent failure the arm was built to stop."""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
out = await _run_tool_arm([], rec, retrieval_log=log)
|
||||
|
||||
@@ -890,10 +877,9 @@ async def test_a_command_the_arm_never_searched_writes_no_row_at_all():
|
||||
def test_no_rule_arm_logs_its_call_behind_a_results_guard():
|
||||
"""Structural, on top of the behavioural tests above, because the defect
|
||||
was one level of indentation and it appeared INDEPENDENTLY in two arms —
|
||||
the pre-tool arm inherited it by being modelled on its sibling (#3497).
|
||||
the pre-tool arm inherited it by being modelled on its sibling.
|
||||
|
||||
Since milestone 456 there is one implementation to guard rather than a
|
||||
copy per arm, so the property is checked where it is written: the
|
||||
There is one implementation to guard rather than a copy per arm, so the property is checked where it is written: the
|
||||
pipeline's ranked-search stage writes the call row unconditionally, the
|
||||
arm runner calls that stage before it can return, and the surfacing row
|
||||
alone stays behind `if rule_ids:`. And no arm writes a rule-source row of
|
||||
@@ -921,7 +907,7 @@ def test_no_rule_arm_logs_its_call_behind_a_results_guard():
|
||||
assert not early, (
|
||||
f"the ranked-search stage can return at line {early[0]} before logging "
|
||||
f"its call — a call that found nothing is the only evidence a bar is "
|
||||
f"too high (#3497)"
|
||||
f"too high"
|
||||
)
|
||||
|
||||
# 2. The runner reaches the stage before its first early return.
|
||||
@@ -958,7 +944,7 @@ def test_no_rule_arm_logs_its_call_behind_a_results_guard():
|
||||
)
|
||||
|
||||
|
||||
# ── Suppression: which zeros were the ranker, which were repeats (#3497) ──
|
||||
# ── Suppression: which zeros were the ranker, which were repeats ──────
|
||||
#
|
||||
# Making the call log unconditional exposed a second ambiguity in the same row.
|
||||
# A zero-result rule call is two unrelated events: the ranker found nothing
|
||||
@@ -1020,19 +1006,15 @@ async def test_a_shown_hit_is_not_counted_as_suppressed():
|
||||
assert len(log.call_args.kwargs["results"]) == 1
|
||||
|
||||
|
||||
# ── The identity that falsified this milestone (#3668) ─────────────────
|
||||
# ── The identity between the two tables ─────────────────────────────────
|
||||
#
|
||||
# `rule_usage.surfaced` == `pre_tool_rule.cleared` + `write_path_rule.cleared`.
|
||||
# Milestone #379 was scoped on a reconstruction that put ~64% of ranked rule
|
||||
# surfacings as never reaching `rule_usage_events`. Five steps were planned
|
||||
# against it. One read of this identity — 17 = 17, then 39 = 39 on a second
|
||||
# window — falsified the whole thing: the gap was two counters that started
|
||||
# recording on different days, not a write path dropping rows.
|
||||
# Two counters that started recording on different days produce a gap that
|
||||
# reads like a write path dropping rows; this identity tells the two apart.
|
||||
#
|
||||
# So the identity is not a nice-to-have. It is the cheapest true statement
|
||||
# available about this pair of tables, and its absence is what let a magnitude
|
||||
# that merely LOOKED wrong survive a code review and a five-step plan. An
|
||||
# identity that must hold exactly beats a magnitude that looks wrong.
|
||||
# available about this pair of tables. An identity that must hold exactly
|
||||
# beats a magnitude that looks wrong.
|
||||
#
|
||||
# WHY THE ARM IS THE RIGHT PLACE TO PIN IT, and the readout is not. Inside an
|
||||
# arm, one `fresh` list feeds both recorders in one function, so the counts
|
||||
@@ -1040,8 +1022,8 @@ async def test_a_shown_hit_is_not_counted_as_suppressed():
|
||||
# than it looks: `cleared_threshold` counts CALLS that beat the bar while
|
||||
# `surfaced` counts RULES, and those coincide only while `RULEHINT_LIMIT` is 1.
|
||||
# Raise the limit and the readout identity breaks while nothing is wrong.
|
||||
# `RULEHINT_LIMIT` has already moved once (2 → 1, `2385100`), and that move is
|
||||
# half of why the original reconstruction misread its own numbers.
|
||||
# `RULEHINT_LIMIT` can move, and a reader comparing windows across a move
|
||||
# misreads its own numbers.
|
||||
#
|
||||
# Hence three hits below, where production currently returns at most one. The
|
||||
# test is deliberately in a state the limit does not permit today, because what
|
||||
@@ -1055,7 +1037,7 @@ _THREE_HITS = [
|
||||
]
|
||||
|
||||
def test_the_three_hit_fixture_sits_inside_the_rule_band():
|
||||
"""The fixture's own precondition, asserted rather than commented (#3851).
|
||||
"""The fixture's own precondition, asserted rather than commented.
|
||||
|
||||
The act arms band before they dedup, so a fixture whose spread straddles
|
||||
`_RULEHINT_BAND` loses its lowest hit to the BAND and then reports a count
|
||||
@@ -1085,8 +1067,7 @@ _ARMS = [
|
||||
("write_path_rule", _run_arm),
|
||||
("pre_tool_rule", _run_tool_arm),
|
||||
# The prompt arm joins the family rather than being modelled on it — see
|
||||
# the block above _prompt_patches for why that distinction is the whole
|
||||
# lesson of #3497.
|
||||
# the block above _prompt_patches for why that distinction matters.
|
||||
("prompt_rule", _run_prompt_arm),
|
||||
]
|
||||
|
||||
@@ -1140,7 +1121,7 @@ async def test_both_recorders_report_the_same_rules_for_one_call(
|
||||
f"call log recorded {logged} and the surfacing log recorded {surfaced}. "
|
||||
f"Both come from `fresh`, in one function, so any difference is a bug "
|
||||
f"in the wiring — and it is the shape that reads as a lost write when "
|
||||
f"the two tables are later compared in aggregate (#3668)."
|
||||
f"the two tables are later compared in aggregate."
|
||||
)
|
||||
assert len(logged) == expected, (
|
||||
"the fixture stopped exercising what it claims to; check the exclusion "
|
||||
@@ -1148,18 +1129,15 @@ async def test_both_recorders_report_the_same_rules_for_one_call(
|
||||
)
|
||||
|
||||
|
||||
# ── A repeat is REFERENCED, not withheld (#3750) ──────────────────────────
|
||||
# ── A repeat is REFERENCED, not withheld ─────────────────────────────────
|
||||
#
|
||||
# Both arms used to drop a hit already on the session's exclusion ledger and
|
||||
# emit nothing at all. That is right only while the session still HOLDS what it
|
||||
# was told — and it stops being right the moment a compaction summarizes the
|
||||
# Dropping a hit already on the session's exclusion ledger and emitting
|
||||
# nothing is right only while the session still HOLDS what it was told — and it stops being right the moment a compaction summarizes the
|
||||
# earlier injection away while the id stays on the ledger, which leaves the
|
||||
# rule absent from context AND unreachable for the rest of the session.
|
||||
#
|
||||
# The tests below pin the emitted LINE, not the prose describing it, and both
|
||||
# arms are parametrized through one body so the two tails cannot drift apart —
|
||||
# #3497's history is that the pre-tool arm inherited a defect from its sibling
|
||||
# by being modelled on it rather than sharing with it.
|
||||
# arms are parametrized through one body so the two tails cannot drift apart.
|
||||
|
||||
_HELD = fake_rule(
|
||||
id=156,
|
||||
@@ -1168,11 +1146,10 @@ _HELD = fake_rule(
|
||||
when_to_apply="writing any call that crosses a process boundary",
|
||||
)
|
||||
_FRESH_TAIL = "not in this session's loaded set"
|
||||
# `seen` WITHOUT `held` is the middle state since #4100: an arm named this rule
|
||||
# earlier, and the session never opened it. Every test below drives the arms
|
||||
# with `exclude_rule_ids` alone — the naming ledger — so this is the tail they
|
||||
# produce, and it is the one that used to claim "You saw it earlier this
|
||||
# session" about a teaser nobody had read.
|
||||
# `seen` WITHOUT `held` is the middle state: an arm named this rule earlier,
|
||||
# and the session never opened it. Every test below drives the arms with
|
||||
# `exclude_rule_ids` alone — the naming ledger — so this is the tail they
|
||||
# produce, and it must not claim the session read a teaser nobody opened.
|
||||
_SEEN_TAIL = "Mentioned earlier this session but not opened"
|
||||
# The third state, reached only when the opened ledger names the rule too.
|
||||
_HELD_TAIL = "You opened it earlier this session"
|
||||
@@ -1181,19 +1158,18 @@ _HELD_TAIL = "You opened it earlier this session"
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_rule_already_on_the_ledger_still_produces_a_line(source, run):
|
||||
"""THE REGRESSION, stated as the thing that used to be absent.
|
||||
"""A repeat still renders a line.
|
||||
|
||||
Falsified against the old behaviour: before #3750 both arms filtered to
|
||||
`fresh` before rendering, so an all-excluded call returned an empty
|
||||
context and this assertion fails on `context == ""`.
|
||||
An arm that filters to `fresh` before rendering returns an empty context
|
||||
for an all-excluded call, and this assertion fails on `context == ""`.
|
||||
"""
|
||||
out = await run([(0.81, _HELD)], MagicMock(), exclude_rule_ids=[156])
|
||||
|
||||
# ON `get_rule(156)` RATHER THAN A TRUTHY CONTEXT. The write-path arm's
|
||||
# context also carries the prior-art menu, the staleness line and the shape
|
||||
# signals, so `assert out["context"]` is TRUE under the old behaviour and
|
||||
# would pin nothing on that arm while looking identical to a real check on
|
||||
# the other. The rule line is the only part of the string this changes.
|
||||
# signals, so `assert out["context"]` is TRUE even when the rule line is
|
||||
# missing, and would pin nothing on that arm while looking identical to a
|
||||
# real check on the other. The rule line is the part under test.
|
||||
assert "get_rule(156)" in out["context"], (
|
||||
f"{source} emitted no rule line for a rule the session had already "
|
||||
f"been shown. Silence is only correct while the session still holds "
|
||||
@@ -1229,14 +1205,14 @@ async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, ru
|
||||
)
|
||||
assert fresh_ctx != seen_ctx, "the two tails collapsed into one"
|
||||
# Neither call named an OPENED ledger, so neither may claim the session
|
||||
# read anything (#4100). Catches `held` defaulting true, which would make
|
||||
# read anything. Catches `held` defaulting true, which would make
|
||||
# every repeat assert the strongest of the three claims on no evidence.
|
||||
assert _HELD_TAIL not in fresh_ctx and _HELD_TAIL not in seen_ctx, (
|
||||
f"{source} said the session had opened a rule it was only shown"
|
||||
)
|
||||
|
||||
|
||||
# ── the third axis: KIND (milestone 399) ────────────────────────────────
|
||||
# ── the third axis: KIND ─────────────────────────────────────────────────
|
||||
#
|
||||
# A preference is a rule row that does not bind, so it rides the same arms and
|
||||
# the same line. What must differ is the REGISTER: a rule's line tells the
|
||||
@@ -1245,8 +1221,8 @@ async def test_the_two_tails_are_distinguishable_and_say_the_true_one(source, ru
|
||||
# to find how this has been done, and following it buys consistency rather
|
||||
# than correctness.
|
||||
#
|
||||
# Rendered in the rule's voice, a preference becomes the thing milestone 399
|
||||
# exists to avoid: a rule with a different column.
|
||||
# Rendered in the rule's voice, a preference becomes a rule with a different
|
||||
# column.
|
||||
|
||||
_PREF = fake_rule(
|
||||
id=177,
|
||||
@@ -1290,7 +1266,7 @@ async def test_kind_and_seen_do_not_read_each_other(source, run):
|
||||
|
||||
Whether a record is already on the exclusion ledger has nothing to do with
|
||||
how much force it carries. Keeping the two unrelated in the code is what
|
||||
let a second kind arrive without reopening #3750's repeat question — and
|
||||
let a second kind arrive without reopening the repeat question — and
|
||||
the way that silently breaks is a `seen` branch that grows a kind test,
|
||||
or a kind branch that grows a `seen` test, leaving one of the four
|
||||
combinations rendered by nobody's intention.
|
||||
@@ -1346,9 +1322,9 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
|
||||
)
|
||||
|
||||
|
||||
# ── What a reference IS in the telemetry: nothing new (#3752) ─────────────
|
||||
# ── What a reference IS in the telemetry: nothing new ──────────────────────
|
||||
#
|
||||
# THE RELATION, STATED BEFORE IT SHIPS. #3750 changes what is RENDERED and
|
||||
# THE RELATION. Rendering a repeat as a reference changes what is RENDERED and
|
||||
# nothing about what is COUNTED:
|
||||
#
|
||||
# result_count counts fresh surfacings — unchanged
|
||||
@@ -1357,13 +1333,12 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
|
||||
#
|
||||
# A reference is a rendering decision, not a retrieval outcome. That answer is
|
||||
# not free: the naive implementation renders repeats by dropping the `fresh`
|
||||
# filter, which takes `suppressed_count` to zero everywhere — and #3739's
|
||||
# near-miss fix identifies repeat-caused zeros by `suppressed_count > 0`, so
|
||||
# the contamination corrected on 2026-09-08 would return by a different route,
|
||||
# in the same field, with the fix still sitting in the code not working.
|
||||
# filter, which takes `suppressed_count` to zero everywhere — and the
|
||||
# near-miss readout identifies repeat-caused zeros by `suppressed_count > 0`,
|
||||
# so every repeat would be re-counted as a genuine ranker rejection.
|
||||
#
|
||||
# #3712 gave a reader `complete_from` for a counter that started late. Nothing
|
||||
# tells a reader a counter's DEFINITION moved. This test is the cheap version
|
||||
# `complete_from` tells a reader a counter started late. Nothing tells a
|
||||
# reader a counter's DEFINITION moved. This test is the cheap version
|
||||
# of that guarantee: the claim "nothing moved" is only worth anything if it is
|
||||
# checkable, so it is asserted rather than described.
|
||||
|
||||
@@ -1371,7 +1346,7 @@ async def test_neither_tail_injects_the_rule_statement(source, run):
|
||||
@pytest.mark.parametrize(("source", "run"), _ARMS, ids=["write_path", "pre_tool", "prompt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_reference_is_rendered_but_not_counted(source, run):
|
||||
"""The counters must read exactly as they did before #3750."""
|
||||
"""Rendering a repeat leaves every counter as it was."""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
out = await run([(0.81, _HELD)], rec, retrieval_log=log,
|
||||
exclude_rule_ids=[156])
|
||||
@@ -1384,7 +1359,7 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
|
||||
f"is not evidence the bar is set correctly."
|
||||
)
|
||||
assert row["suppressed"] == 1, (
|
||||
f"{source} stopped reporting the repeat as suppressed. #3739's "
|
||||
f"{source} stopped reporting the repeat as suppressed. The "
|
||||
f"near_misses predicate excludes declines with suppressed_count > 0 — "
|
||||
f"if this reads 0, every repeat-caused zero is re-counted as a genuine "
|
||||
f"ranker rejection and the near-miss contamination returns."
|
||||
@@ -1406,7 +1381,7 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
|
||||
)
|
||||
|
||||
|
||||
# ── the reserved preference slot (#3894) ────────────────────────────────
|
||||
# ── the reserved preference slot ────────────────────────────────────────
|
||||
#
|
||||
# A rule and a preference are not equally served by one ranking, because their
|
||||
# losses are not equal. A rule crowded out at the prompt boundary still fires
|
||||
@@ -1416,9 +1391,9 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
|
||||
#
|
||||
# The failure is invisible without this slot. The rule that won is a
|
||||
# legitimate hit, the telemetry reads healthy, and the only symptom is a
|
||||
# preference that quietly never arrives — which is `reuse_slot`'s shape one
|
||||
# corpus over (#2463), where snippets kept losing to project records that
|
||||
# merely resembled the query.
|
||||
# preference that quietly never arrives — `reuse_slot`'s shape one corpus
|
||||
# over, where snippets lose to project records that merely resemble the
|
||||
# query.
|
||||
|
||||
|
||||
def _search_by_kind(general, preference):
|
||||
@@ -1537,9 +1512,8 @@ async def test_the_slot_extends_rather_than_displacing():
|
||||
`reuse_slot` evicts its menu's weakest hit; this one does not, and the
|
||||
reason is the ledger. A displaced hit sits in `prompt_rule`'s
|
||||
retrieval_logs row while never being surfaced, so that source's two
|
||||
tables stop agreeing — and #3668's identity is the cheapest true
|
||||
statement available about this pair. Milestone #379 is what losing it
|
||||
costs: five steps planned against two counters disagreeing.
|
||||
tables stop agreeing — and the identity is the cheapest true statement
|
||||
available about this pair.
|
||||
"""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
out, _ = await _run_slot(
|
||||
@@ -1562,7 +1536,7 @@ async def test_the_slot_extends_rather_than_displacing():
|
||||
]
|
||||
assert logged == surfaced, (
|
||||
f"prompt_rule logged {logged} and surfaced {surfaced} — the identity "
|
||||
f"#3668 pins, broken by the slot rather than by a write path"
|
||||
f"pinned above, broken by the slot rather than by a write path"
|
||||
)
|
||||
|
||||
|
||||
@@ -1570,9 +1544,9 @@ async def test_the_slot_extends_rather_than_displacing():
|
||||
async def test_the_slot_accounts_for_itself_under_its_own_source():
|
||||
"""Both sides of the trade, logged.
|
||||
|
||||
#2463's own finding is the warning rather than the precedent: the hit
|
||||
that slot pushed OUT was in retrieval_logs while the query that pushed it
|
||||
out was not, so the slot could never be judged against what it displaced.
|
||||
A slot whose query is not logged cannot be judged against what it
|
||||
displaced: the hit it pushed OUT is in retrieval_logs and the query that
|
||||
pushed it out is not.
|
||||
This one logs its query AND records its surfacing, under a source of its
|
||||
own, so it can be evaluated separately from the ranking it bypassed.
|
||||
"""
|
||||
@@ -1597,9 +1571,9 @@ async def test_a_preference_on_the_ledger_keeps_the_slot_and_is_not_recounted():
|
||||
|
||||
A preference is the kind of record where being reminded is the point, so
|
||||
one already on the ledger occupies the slot rather than being skipped for
|
||||
a fresh one — rendered with the repeat tail (#3750). What it must not do
|
||||
is register a second surfacing, which would count one delivery twice in
|
||||
the denominator pull-through is read from (#3752).
|
||||
a fresh one — rendered with the repeat tail. What it must not do is
|
||||
register a second surfacing, which would count one delivery twice in the
|
||||
denominator pull-through is read from.
|
||||
"""
|
||||
log, rec = MagicMock(), MagicMock()
|
||||
out, _ = await _run_slot(
|
||||
@@ -1616,19 +1590,17 @@ async def test_a_preference_on_the_ledger_keeps_the_slot_and_is_not_recounted():
|
||||
)
|
||||
assert 140 not in out["rule_ids"], (
|
||||
"a repeat was written back to the hook's ledger, which would keep "
|
||||
"pushing its stamp forward so it never aged out (#3751)"
|
||||
"pushing its stamp forward so it never aged out"
|
||||
)
|
||||
|
||||
|
||||
# ── each act arm uses its OWN bar, end to end (#3853) ───────────────────
|
||||
# ── each act arm uses its OWN bar, end to end ───────────────────────────
|
||||
#
|
||||
# The two act arms shared one threshold until the telemetry showed them
|
||||
# behaving like different subsystems at the same number: write_path_rule
|
||||
# speaking on 37% of 2,325 calls, pre_tool_rule on 2% of 11,768, because a
|
||||
# The two act arms behave like different subsystems at the same number: a
|
||||
# code payload is long and rich where a shell command is short and carries
|
||||
# less signal for the same relevance.
|
||||
#
|
||||
# Splitting the bar creates a failure the old single-bar code could not have:
|
||||
# Separate bars create a failure a single bar could not have:
|
||||
# an arm can now search at one threshold and REPORT another. That row is what
|
||||
# near-miss analysis is read against, so a mismatch does not look like a bug —
|
||||
# it looks like a corpus whose scores sit somewhere they do not, and it would
|
||||
@@ -1652,7 +1624,7 @@ async def test_each_act_arm_searches_at_its_own_bar():
|
||||
|
||||
assert search.await_args.kwargs["threshold"] == 0.61, (
|
||||
"the command arm searched at the write-path arm's bar; the two were "
|
||||
"split at #3853 precisely because one number cannot serve both"
|
||||
"split precisely because one number cannot serve both"
|
||||
)
|
||||
|
||||
|
||||
@@ -1685,11 +1657,11 @@ async def test_an_act_arm_reports_the_bar_it_actually_searched_at():
|
||||
assert rows[0].kwargs["threshold"] == search.await_args.kwargs["threshold"]
|
||||
|
||||
|
||||
# ── scope: a session gets global rules plus its own project's (milestone 414) ──
|
||||
# ── scope: a session gets global rules plus its own project's ──
|
||||
|
||||
|
||||
def test_every_hook_rule_search_says_which_project_it_is_for():
|
||||
"""Every call site passes `project_id`, walked rather than grepped (rule 167).
|
||||
"""Every call site passes `project_id`, walked rather than grepped.
|
||||
|
||||
semantic_search_rules defaults to GLOBAL rules only, so an arm that forgets
|
||||
the keyword does not leak another project's rules — it quietly stops
|
||||
@@ -1698,7 +1670,7 @@ def test_every_hook_rule_search_says_which_project_it_is_for():
|
||||
`everywhere` is not an acceptable answer in a hook, which speaks unasked.
|
||||
"""
|
||||
sources = {
|
||||
# Since milestone 456 the hook arms search through the pipeline's one
|
||||
# The hook arms search through the pipeline's one
|
||||
# ranked-search stage, checked below — so a direct search appearing
|
||||
# here again is a copy of the arm coming back, and fails the count.
|
||||
"src/scribe/services/plugin_context.py": 0,
|
||||
|
||||
Reference in New Issue
Block a user