feat(500): retire the fixed-question preference arm - completion-report preferences ride the moments
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 31s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 31s
report_preference searched one constant string when a task closed and handed matches back as update_task's reply_preferences. Milestone 500 step 3 delivers the reply shapes and the preferences mounted beside them on the moments, so the arm goes, whole (rule 22): - services/reply_preferences.py, the REPORT_PREFERENCE RuleArm, its re-scorer and corpus entries, and the REPORTPREF constants - update_task's reply_preferences block and its cue; the docstring now points at moment_rules - the fixed_query field and the fixed_query_never_clears warning: this arm was the only one that set it, so the concept and the cannot_decline exemption go with it - the two Settings fields for its floor and budget - migration 0121 deletes its rows (logs, judgments, rule-usage events, tuning history, settings keys), operator-approved 2026-10-09. Without it the readout would call the source unregistered and its surfacings would count as ambient. The unindexed rule_usage delete is bounded by created_at. #5496 (step 4 of milestone 500). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -471,83 +471,6 @@ def test_a_quiet_arm_is_not_suspended_either_way() -> None:
|
||||
assert "floor_moved_mid_window" not in codes(ws)
|
||||
|
||||
|
||||
# ── a fixed-query arm: decline rate is arithmetic, not evidence (#4232) ─────
|
||||
#
|
||||
# `report_preference` searches one constant string (COMPLETION_QUERY), so it
|
||||
# scores against one number on every call. Its decline rate is therefore 0% or
|
||||
# 100% and never in between, and which one depends only on where the bar sits
|
||||
# relative to that constant.
|
||||
#
|
||||
# So the two warnings swap roles for these arms. "Never declined" stops being
|
||||
# evidence about the floor — `cannot_decline`'s own remedy, "check that it
|
||||
# applies its floor", is unanswerable from it. "Always declined" starts being
|
||||
# evidence, because for a constant score it means the bar is above it and no
|
||||
# further traffic will ever say otherwise.
|
||||
|
||||
|
||||
def test_cannot_decline_is_silent_on_a_fixed_query_arm():
|
||||
"""The live readout fired this on `report_preference` at 45 calls, 0
|
||||
empty, with p10 = p50 = p90 = min = max = 0.791 — five identical
|
||||
percentiles, which is one record at one score rather than a ranking."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=0, p10=0.791)})
|
||||
assert "cannot_decline" not in codes(ws, "report_preference")
|
||||
|
||||
|
||||
def test_cannot_decline_still_fires_where_the_rate_means_something():
|
||||
"""The falsifier for the case above (rule 167). If this passes only
|
||||
because the check was disabled rather than narrowed, this fails."""
|
||||
ws = warn({"auto_inject": src(calls=N, zero_result_calls=0)})
|
||||
assert "cannot_decline" in codes(ws, "auto_inject")
|
||||
|
||||
|
||||
def test_a_fixed_query_arm_that_never_clears_its_bar_is_named():
|
||||
"""69 consecutive declines at 0.0006 under the bar is a state this arm has
|
||||
actually been in. Nothing else in the readout would have said so: it looks
|
||||
exactly like an arm with nothing to report."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=45)})
|
||||
assert "fixed_query_never_clears" in codes(ws, "report_preference")
|
||||
|
||||
detail = next(w["detail"] for w in ws if w["code"] == "fixed_query_never_clears")
|
||||
assert "near_miss_samples" in detail, (
|
||||
"the last time this fired, every percentile said lower the floor and "
|
||||
"the refused record showed the refusal was right — so the warning has "
|
||||
"to send the reader to the record, not to the dial"
|
||||
)
|
||||
|
||||
|
||||
def test_an_ordinary_arm_returning_nothing_all_window_is_not_dead():
|
||||
"""For an arm whose score can vary, an empty window means nothing matched,
|
||||
which is an answer rather than a fault."""
|
||||
ws = warn({"auto_inject": src(calls=N, zero_result_calls=N)})
|
||||
assert "fixed_query_never_clears" not in codes(ws, "auto_inject")
|
||||
|
||||
|
||||
def test_a_fixed_query_arm_that_sometimes_clears_is_not_dead():
|
||||
"""Only ALL-empty says the bar is above the constant. Anything in between
|
||||
means the score is not actually constant, and the premise is wrong."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=44)})
|
||||
assert "fixed_query_never_clears" not in codes(ws, "report_preference")
|
||||
|
||||
|
||||
def test_the_dead_arm_warning_still_needs_volume():
|
||||
ws = warn({"report_preference": src(calls=N - 1, zero_result_calls=N - 1)})
|
||||
assert "fixed_query_never_clears" not in codes(ws)
|
||||
|
||||
|
||||
def test_the_registry_declares_which_arms_ask_a_fixed_question():
|
||||
"""Asserted on structure (rule 167), and able to fail: if `fixed_query`
|
||||
is dropped or defaults to True, one of these two halves breaks."""
|
||||
from scribe.services.retrieval_registry import POINTS
|
||||
|
||||
assert POINTS["report_preference"].fixed_query is True, (
|
||||
"services/reply_preferences.py::COMPLETION_QUERY is a module constant"
|
||||
)
|
||||
# An arm whose query is built from the prompt, the file or the command is
|
||||
# not fixed, and marking one would silence a warning that works there.
|
||||
for varying in ("auto_inject", "write_path", "pre_tool_rule", "prompt_rule"):
|
||||
assert POINTS[varying].fixed_query is False, varying
|
||||
|
||||
|
||||
# ── floor_history_unknown: the window reaches back past the ledger ─────────
|
||||
#
|
||||
# THE SAME SUSPENSION, FOR THE CASE THE ONE ABOVE CANNOT SEE.
|
||||
|
||||
Reference in New Issue
Block a user