Retrieval: the passage that matched, every kind searchable, work logs and charters findable #175

Merged
bvandeusen merged 9 commits from dev into main 2026-09-21 12:39:19 -04:00
4 changed files with 169 additions and 2 deletions
Showing only changes of commit f8e53c1c35 - Show all commits
+13 -1
View File
@@ -421,7 +421,19 @@ It is an UPPER BOUND per surface: a pull records the door it came
Only ever raised for unbidden arms known to log unconditionally: a
search returning a list every time is doing its job, and an arm whose
zeros were never written would flag a LOGGING bug while pointing you at
a threshold, which is #3497 exactly.
a threshold, which is #3497 exactly. Nor for an arm whose query never
changes — see the next entry.
- `fixed_query_never_clears` — an arm that always searches the SAME query
returned nothing on every call. Its score is one constant, so this is
not a quiet window: the bar sits above that constant and no amount of
further traffic will produce a different result. The arm is off rather
than silent, and nothing else here would say so. The same property is
why `cannot_decline` is not raised for these arms: with a constant
score the decline rate is 0% or 100% by construction, so "never
declined" is arithmetic and not evidence about the floor. Read the
refused record (`near_miss_samples`) BEFORE moving the dial — the last
time an arm sat here, every percentile said lower it and the refused
record showed the refusal was right.
- `band_hugs_floor` — the weakest tenth of what an arm returns sits on
its floor. The bar is doing the selecting and the score is not, so
moving that floor changes how MUCH you get, not how good it is.
+29 -1
View File
@@ -86,6 +86,29 @@ class Point:
quiet_because: str = ""
fixed_query: bool = False
"""Whether this arm always searches the SAME query string.
THE #3497 GUARD, ONE STEP OVER. `logs_unconditionally` below exists
because a warning computed over a LOGGING property read as a ranking
problem. This field exists because a warning computed over a QUERY-SHAPE
property does the same thing.
An arm with a fixed query scores against one constant. Its decline rate is
therefore 0% or 100% and nothing in between — which of the two depends
only on whether the bar sits below or above that single number. So "never
returned nothing" says nothing at all about whether a floor is applied,
and `cannot_decline` — whose whole remedy is "check that it applies its
floor" — is uninformative here and skips these arms.
What IS informative for them is the mirror image, and `reply_preferences`
names it in its own docstring: every call returning nothing means the bar
sits above the constant, no traffic will ever move it, and the arm is
dead. That has happened — 69 consecutive declines at 0.0006 under the bar
(see `retrieval_surfaces`) — so it gets its own warning rather than
inheriting one written for arms whose score can vary.
"""
logs_unconditionally: bool = True
"""Whether this arm writes a row even when it returns NOTHING.
@@ -114,8 +137,13 @@ POINTS: dict[str, Point] = dict([
"the one line reserved for a preference at the prompt boundary"),
_p("reuse_slot", UNBIDDEN, "the one line reserved for a reusable snippet"),
_p("lesson_slot", UNBIDDEN, "the one line reserved for a lesson"),
# `fixed_query`: COMPLETION_QUERY is a module constant in
# services/reply_preferences.py, so this arm's top score is the same number
# on every call — measured at 0.791 across 45 consecutive calls, with p10,
# p50, p90, min and max all identical. Five equal percentiles is the tell.
_p("report_preference", UNBIDDEN,
"the fixed question asked when a task finishes: how should this report read"),
"the fixed question asked when a task finishes: how should this report read",
fixed_query=True),
# ── Asked: a caller wanted a ranked list ─────────────────────────────
_p("mcp_search", ASKED, "an agent called search"),
@@ -556,12 +556,22 @@ def _compute_warnings(sources: dict, usage: dict, rule_usage: dict,
# arms once recorded only their hits, so their decline count was
# structurally zero and this warning would have fired on a LOGGING
# defect while pointing the reader at the threshold.
#
# FOUR NOW. A FIXED-QUERY arm is exempt for the same reason one step
# over (#4232): it scores against one constant, so its decline rate is
# 0% or 100% and never in between, and which one it is depends only on
# where the bar sits relative to that single number. "Never returned
# nothing" is then not evidence about the floor — it is arithmetic —
# and this warning's own remedy, "check that it applies its floor",
# cannot be answered from it. Those arms get `fixed_query_never_clears`
# below, which asks the question that IS answerable for them.
if (
calls >= min_calls
and (b.get("zero_result_calls") or 0) == 0
and point is not None
and point.kind == UNBIDDEN
and point.logs_unconditionally
and not point.fixed_query
):
out.append(_warn(
"cannot_decline",
@@ -572,6 +582,46 @@ def _compute_warnings(sources: dict, usage: dict, rule_usage: dict,
source=name, calls=calls, zero_result_calls=0,
))
# ── A fixed-query arm that never clears its bar ──────────────────
#
# The mirror image of `cannot_decline`, and the state that actually
# threatens these arms. `reply_preferences` names it in its own
# docstring: "a fixed query makes this arm's score a constant and a
# floor a hair above it produces a dead arm no amount of traffic will
# ever reveal."
#
# For an arm whose score can vary, a window of all-empty calls is
# ordinary — it means nothing matched, which is an answer. For one
# whose score is a constant it means the bar is above that constant,
# and no volume of further calls will ever produce a different result.
# The arm is not quiet; it is switched off, and nothing else in this
# readout would say so.
#
# It has happened: `report_preference` logged 69 consecutive declines
# at 0.0006 under the bar. Note what that incident also proves — the
# fix is NOT automatically to lower the floor. Reading the refused
# record showed the refusal was correct, so this warning sends the
# reader to `near_miss_samples` rather than to the dial.
if (
calls >= min_calls
and point is not None
and point.fixed_query
and point.logs_unconditionally
and (b.get("zero_result_calls") or 0) == calls
):
out.append(_warn(
"fixed_query_never_clears",
f"{calls} calls, every one of them empty — and this arm always "
f"searches the same query, so its score is a constant. That "
f"means the bar sits above it and no amount of further traffic "
f"will change the result: the arm is off, not quiet. Read the "
f"record it refused (`near_miss_samples`) before touching the "
f"floor — the last time this arm sat here, every percentile "
f"said lower it and the refused record showed the refusal was "
f"right.",
source=name, calls=calls, zero_result_calls=calls,
))
# ── Band hugs its floor ──────────────────────────────────────────
#
# Read on p10, the WEAKEST tenth of what the arm returned. If even
+77
View File
@@ -446,3 +446,80 @@ def test_a_quiet_arm_is_not_suspended_either_way() -> None:
ws = warn({"auto_inject": src(calls=1, zero_result_calls=0, p10=0.705)},
floors={"auto_inject": 0.70}, floor_moves={"auto_inject": MOVED})
assert "floor_moved_mid_window" not in codes(ws)
# ── a fixed-query arm: decline rate is arithmetic, not evidence (#4232) ─────
#
# `report_preference` searches one constant string (COMPLETION_QUERY), so it
# scores against one number on every call. Its decline rate is therefore 0% or
# 100% and never in between, and which one depends only on where the bar sits
# relative to that constant.
#
# So the two warnings swap roles for these arms. "Never declined" stops being
# evidence about the floor — `cannot_decline`'s own remedy, "check that it
# applies its floor", is unanswerable from it. "Always declined" starts being
# evidence, because for a constant score it means the bar is above it and no
# further traffic will ever say otherwise.
def test_cannot_decline_is_silent_on_a_fixed_query_arm():
"""The live readout fired this on `report_preference` at 45 calls, 0
empty, with p10 = p50 = p90 = min = max = 0.791 — five identical
percentiles, which is one record at one score rather than a ranking."""
ws = warn({"report_preference": src(calls=45, zero_result_calls=0, p10=0.791)})
assert "cannot_decline" not in codes(ws, "report_preference")
def test_cannot_decline_still_fires_where_the_rate_means_something():
"""The falsifier for the case above (rule 167). If this passes only
because the check was disabled rather than narrowed, this fails."""
ws = warn({"auto_inject": src(calls=N, zero_result_calls=0)})
assert "cannot_decline" in codes(ws, "auto_inject")
def test_a_fixed_query_arm_that_never_clears_its_bar_is_named():
"""69 consecutive declines at 0.0006 under the bar is a state this arm has
actually been in. Nothing else in the readout would have said so: it looks
exactly like an arm with nothing to report."""
ws = warn({"report_preference": src(calls=45, zero_result_calls=45)})
assert "fixed_query_never_clears" in codes(ws, "report_preference")
detail = next(w["detail"] for w in ws if w["code"] == "fixed_query_never_clears")
assert "near_miss_samples" in detail, (
"the last time this fired, every percentile said lower the floor and "
"the refused record showed the refusal was right — so the warning has "
"to send the reader to the record, not to the dial"
)
def test_an_ordinary_arm_returning_nothing_all_window_is_not_dead():
"""For an arm whose score can vary, an empty window means nothing matched,
which is an answer rather than a fault."""
ws = warn({"auto_inject": src(calls=N, zero_result_calls=N)})
assert "fixed_query_never_clears" not in codes(ws, "auto_inject")
def test_a_fixed_query_arm_that_sometimes_clears_is_not_dead():
"""Only ALL-empty says the bar is above the constant. Anything in between
means the score is not actually constant, and the premise is wrong."""
ws = warn({"report_preference": src(calls=45, zero_result_calls=44)})
assert "fixed_query_never_clears" not in codes(ws, "report_preference")
def test_the_dead_arm_warning_still_needs_volume():
ws = warn({"report_preference": src(calls=N - 1, zero_result_calls=N - 1)})
assert "fixed_query_never_clears" not in codes(ws)
def test_the_registry_declares_which_arms_ask_a_fixed_question():
"""Asserted on structure (rule 167), and able to fail: if `fixed_query`
is dropped or defaults to True, one of these two halves breaks."""
from scribe.services.retrieval_registry import POINTS
assert POINTS["report_preference"].fixed_query is True, (
"services/reply_preferences.py::COMPLETION_QUERY is a module constant"
)
# An arm whose query is built from the prompt, the file or the command is
# not fixed, and marking one would silence a warning that works there.
for varying in ("auto_inject", "write_path", "pre_tool_rule", "prompt_rule"):
assert POINTS[varying].fixed_query is False, varying