Retrieval: the passage that matched, every kind searchable, work logs and charters findable #175
@@ -421,7 +421,19 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
Only ever raised for unbidden arms known to log unconditionally: a
|
||||
search returning a list every time is doing its job, and an arm whose
|
||||
zeros were never written would flag a LOGGING bug while pointing you at
|
||||
a threshold, which is #3497 exactly.
|
||||
a threshold, which is #3497 exactly. Nor for an arm whose query never
|
||||
changes — see the next entry.
|
||||
- `fixed_query_never_clears` — an arm that always searches the SAME query
|
||||
returned nothing on every call. Its score is one constant, so this is
|
||||
not a quiet window: the bar sits above that constant and no amount of
|
||||
further traffic will produce a different result. The arm is off rather
|
||||
than silent, and nothing else here would say so. The same property is
|
||||
why `cannot_decline` is not raised for these arms: with a constant
|
||||
score the decline rate is 0% or 100% by construction, so "never
|
||||
declined" is arithmetic and not evidence about the floor. Read the
|
||||
refused record (`near_miss_samples`) BEFORE moving the dial — the last
|
||||
time an arm sat here, every percentile said lower it and the refused
|
||||
record showed the refusal was right.
|
||||
- `band_hugs_floor` — the weakest tenth of what an arm returns sits on
|
||||
its floor. The bar is doing the selecting and the score is not, so
|
||||
moving that floor changes how MUCH you get, not how good it is.
|
||||
|
||||
@@ -86,6 +86,29 @@ class Point:
|
||||
|
||||
quiet_because: str = ""
|
||||
|
||||
fixed_query: bool = False
|
||||
"""Whether this arm always searches the SAME query string.
|
||||
|
||||
THE #3497 GUARD, ONE STEP OVER. `logs_unconditionally` below exists
|
||||
because a warning computed over a LOGGING property read as a ranking
|
||||
problem. This field exists because a warning computed over a QUERY-SHAPE
|
||||
property does the same thing.
|
||||
|
||||
An arm with a fixed query scores against one constant. Its decline rate is
|
||||
therefore 0% or 100% and nothing in between — which of the two depends
|
||||
only on whether the bar sits below or above that single number. So "never
|
||||
returned nothing" says nothing at all about whether a floor is applied,
|
||||
and `cannot_decline` — whose whole remedy is "check that it applies its
|
||||
floor" — is uninformative here and skips these arms.
|
||||
|
||||
What IS informative for them is the mirror image, and `reply_preferences`
|
||||
names it in its own docstring: every call returning nothing means the bar
|
||||
sits above the constant, no traffic will ever move it, and the arm is
|
||||
dead. That has happened — 69 consecutive declines at 0.0006 under the bar
|
||||
(see `retrieval_surfaces`) — so it gets its own warning rather than
|
||||
inheriting one written for arms whose score can vary.
|
||||
"""
|
||||
|
||||
logs_unconditionally: bool = True
|
||||
"""Whether this arm writes a row even when it returns NOTHING.
|
||||
|
||||
@@ -114,8 +137,13 @@ POINTS: dict[str, Point] = dict([
|
||||
"the one line reserved for a preference at the prompt boundary"),
|
||||
_p("reuse_slot", UNBIDDEN, "the one line reserved for a reusable snippet"),
|
||||
_p("lesson_slot", UNBIDDEN, "the one line reserved for a lesson"),
|
||||
# `fixed_query`: COMPLETION_QUERY is a module constant in
|
||||
# services/reply_preferences.py, so this arm's top score is the same number
|
||||
# on every call — measured at 0.791 across 45 consecutive calls, with p10,
|
||||
# p50, p90, min and max all identical. Five equal percentiles is the tell.
|
||||
_p("report_preference", UNBIDDEN,
|
||||
"the fixed question asked when a task finishes: how should this report read"),
|
||||
"the fixed question asked when a task finishes: how should this report read",
|
||||
fixed_query=True),
|
||||
|
||||
# ── Asked: a caller wanted a ranked list ─────────────────────────────
|
||||
_p("mcp_search", ASKED, "an agent called search"),
|
||||
|
||||
@@ -556,12 +556,22 @@ def _compute_warnings(sources: dict, usage: dict, rule_usage: dict,
|
||||
# arms once recorded only their hits, so their decline count was
|
||||
# structurally zero and this warning would have fired on a LOGGING
|
||||
# defect while pointing the reader at the threshold.
|
||||
#
|
||||
# FOUR NOW. A FIXED-QUERY arm is exempt for the same reason one step
|
||||
# over (#4232): it scores against one constant, so its decline rate is
|
||||
# 0% or 100% and never in between, and which one it is depends only on
|
||||
# where the bar sits relative to that single number. "Never returned
|
||||
# nothing" is then not evidence about the floor — it is arithmetic —
|
||||
# and this warning's own remedy, "check that it applies its floor",
|
||||
# cannot be answered from it. Those arms get `fixed_query_never_clears`
|
||||
# below, which asks the question that IS answerable for them.
|
||||
if (
|
||||
calls >= min_calls
|
||||
and (b.get("zero_result_calls") or 0) == 0
|
||||
and point is not None
|
||||
and point.kind == UNBIDDEN
|
||||
and point.logs_unconditionally
|
||||
and not point.fixed_query
|
||||
):
|
||||
out.append(_warn(
|
||||
"cannot_decline",
|
||||
@@ -572,6 +582,46 @@ def _compute_warnings(sources: dict, usage: dict, rule_usage: dict,
|
||||
source=name, calls=calls, zero_result_calls=0,
|
||||
))
|
||||
|
||||
# ── A fixed-query arm that never clears its bar ──────────────────
|
||||
#
|
||||
# The mirror image of `cannot_decline`, and the state that actually
|
||||
# threatens these arms. `reply_preferences` names it in its own
|
||||
# docstring: "a fixed query makes this arm's score a constant and a
|
||||
# floor a hair above it produces a dead arm no amount of traffic will
|
||||
# ever reveal."
|
||||
#
|
||||
# For an arm whose score can vary, a window of all-empty calls is
|
||||
# ordinary — it means nothing matched, which is an answer. For one
|
||||
# whose score is a constant it means the bar is above that constant,
|
||||
# and no volume of further calls will ever produce a different result.
|
||||
# The arm is not quiet; it is switched off, and nothing else in this
|
||||
# readout would say so.
|
||||
#
|
||||
# It has happened: `report_preference` logged 69 consecutive declines
|
||||
# at 0.0006 under the bar. Note what that incident also proves — the
|
||||
# fix is NOT automatically to lower the floor. Reading the refused
|
||||
# record showed the refusal was correct, so this warning sends the
|
||||
# reader to `near_miss_samples` rather than to the dial.
|
||||
if (
|
||||
calls >= min_calls
|
||||
and point is not None
|
||||
and point.fixed_query
|
||||
and point.logs_unconditionally
|
||||
and (b.get("zero_result_calls") or 0) == calls
|
||||
):
|
||||
out.append(_warn(
|
||||
"fixed_query_never_clears",
|
||||
f"{calls} calls, every one of them empty — and this arm always "
|
||||
f"searches the same query, so its score is a constant. That "
|
||||
f"means the bar sits above it and no amount of further traffic "
|
||||
f"will change the result: the arm is off, not quiet. Read the "
|
||||
f"record it refused (`near_miss_samples`) before touching the "
|
||||
f"floor — the last time this arm sat here, every percentile "
|
||||
f"said lower it and the refused record showed the refusal was "
|
||||
f"right.",
|
||||
source=name, calls=calls, zero_result_calls=calls,
|
||||
))
|
||||
|
||||
# ── Band hugs its floor ──────────────────────────────────────────
|
||||
#
|
||||
# Read on p10, the WEAKEST tenth of what the arm returned. If even
|
||||
|
||||
@@ -446,3 +446,80 @@ def test_a_quiet_arm_is_not_suspended_either_way() -> None:
|
||||
ws = warn({"auto_inject": src(calls=1, zero_result_calls=0, p10=0.705)},
|
||||
floors={"auto_inject": 0.70}, floor_moves={"auto_inject": MOVED})
|
||||
assert "floor_moved_mid_window" not in codes(ws)
|
||||
|
||||
|
||||
# ── a fixed-query arm: decline rate is arithmetic, not evidence (#4232) ─────
|
||||
#
|
||||
# `report_preference` searches one constant string (COMPLETION_QUERY), so it
|
||||
# scores against one number on every call. Its decline rate is therefore 0% or
|
||||
# 100% and never in between, and which one depends only on where the bar sits
|
||||
# relative to that constant.
|
||||
#
|
||||
# So the two warnings swap roles for these arms. "Never declined" stops being
|
||||
# evidence about the floor — `cannot_decline`'s own remedy, "check that it
|
||||
# applies its floor", is unanswerable from it. "Always declined" starts being
|
||||
# evidence, because for a constant score it means the bar is above it and no
|
||||
# further traffic will ever say otherwise.
|
||||
|
||||
|
||||
def test_cannot_decline_is_silent_on_a_fixed_query_arm():
|
||||
"""The live readout fired this on `report_preference` at 45 calls, 0
|
||||
empty, with p10 = p50 = p90 = min = max = 0.791 — five identical
|
||||
percentiles, which is one record at one score rather than a ranking."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=0, p10=0.791)})
|
||||
assert "cannot_decline" not in codes(ws, "report_preference")
|
||||
|
||||
|
||||
def test_cannot_decline_still_fires_where_the_rate_means_something():
|
||||
"""The falsifier for the case above (rule 167). If this passes only
|
||||
because the check was disabled rather than narrowed, this fails."""
|
||||
ws = warn({"auto_inject": src(calls=N, zero_result_calls=0)})
|
||||
assert "cannot_decline" in codes(ws, "auto_inject")
|
||||
|
||||
|
||||
def test_a_fixed_query_arm_that_never_clears_its_bar_is_named():
|
||||
"""69 consecutive declines at 0.0006 under the bar is a state this arm has
|
||||
actually been in. Nothing else in the readout would have said so: it looks
|
||||
exactly like an arm with nothing to report."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=45)})
|
||||
assert "fixed_query_never_clears" in codes(ws, "report_preference")
|
||||
|
||||
detail = next(w["detail"] for w in ws if w["code"] == "fixed_query_never_clears")
|
||||
assert "near_miss_samples" in detail, (
|
||||
"the last time this fired, every percentile said lower the floor and "
|
||||
"the refused record showed the refusal was right — so the warning has "
|
||||
"to send the reader to the record, not to the dial"
|
||||
)
|
||||
|
||||
|
||||
def test_an_ordinary_arm_returning_nothing_all_window_is_not_dead():
|
||||
"""For an arm whose score can vary, an empty window means nothing matched,
|
||||
which is an answer rather than a fault."""
|
||||
ws = warn({"auto_inject": src(calls=N, zero_result_calls=N)})
|
||||
assert "fixed_query_never_clears" not in codes(ws, "auto_inject")
|
||||
|
||||
|
||||
def test_a_fixed_query_arm_that_sometimes_clears_is_not_dead():
|
||||
"""Only ALL-empty says the bar is above the constant. Anything in between
|
||||
means the score is not actually constant, and the premise is wrong."""
|
||||
ws = warn({"report_preference": src(calls=45, zero_result_calls=44)})
|
||||
assert "fixed_query_never_clears" not in codes(ws, "report_preference")
|
||||
|
||||
|
||||
def test_the_dead_arm_warning_still_needs_volume():
|
||||
ws = warn({"report_preference": src(calls=N - 1, zero_result_calls=N - 1)})
|
||||
assert "fixed_query_never_clears" not in codes(ws)
|
||||
|
||||
|
||||
def test_the_registry_declares_which_arms_ask_a_fixed_question():
|
||||
"""Asserted on structure (rule 167), and able to fail: if `fixed_query`
|
||||
is dropped or defaults to True, one of these two halves breaks."""
|
||||
from scribe.services.retrieval_registry import POINTS
|
||||
|
||||
assert POINTS["report_preference"].fixed_query is True, (
|
||||
"services/reply_preferences.py::COMPLETION_QUERY is a module constant"
|
||||
)
|
||||
# An arm whose query is built from the prompt, the file or the command is
|
||||
# not fixed, and marking one would silence a warning that works there.
|
||||
for varying in ("auto_inject", "write_path", "pre_tool_rule", "prompt_rule"):
|
||||
assert POINTS[varying].fixed_query is False, varying
|
||||
|
||||
Reference in New Issue
Block a user