feat(retrieval): the wide net becomes a pull — fifty candidates, no bar (#4103)

Milestone 416 step 5. The operator's compromise — "if you're worried
about excluding potentially important data let limit it to 50 entries"
— moved to the surface where it is safe. Fifty in the push would be
milestone 394 with extra steps; fifty in a pull crowds nothing out.

`what_might_apply(query)` returns ranked rule candidates with NO
threshold. The moment it serves is the one where the caller does not
trust a bar to decide for them, so it does not have one — every row
carries its score and the reader judges.

WHY IT IS NOT A BIGGER `limit` ON `search`

`_search_rules` returns statement, why and how_to_apply in full, on the
stated reasoning that a caller who went looking deserves the whole
record. That is the DEEP pull and should stay that way. This is the
SHALLOW one — many candidates, each just enough to decide whether to
open it. Opposite trade-offs, so it is a second tool.

THE PREMISE THE STEP GOT WRONG

The task said fifty "costs nothing". `_rule_hint_line` had already
measured otherwise: ~143 tokens per line once the trigger is rendered,
and #3855 tripled trigger lengths across the corpus. Fifty is ~7,000
tokens — cheap next to an arm firing before every Bash call, but not
free, and a tool promising a free wide net gets reached for casually
and then regretted.

So it reuses the graduated shape #3851 measured for the push: the top
few carry their trigger whole, the rest carry a cut of it. TRUNCATED,
never dropped — the trigger is what lets a reader judge without
opening, and a teaser without one is just an id. The cut borrows
`_goal_line`'s technique including the fallback that matters (#4036):
`textwrap.shorten` returns a bare "…" for one unbroken word.

TELEMETRY

Logged under its own pull source, asserted absent from both the tunable
push registry and AMBIENT_SOURCES. That guard is load-bearing right
now: the push arms' near-miss distributions are the evidence #4121
argues from, and a pull folded into them would move those numbers.

INSTRUCTION SURFACES

The using-scribe reflex and the MCP instructions both pointed at
`search(content_type="rule")` for the consequential moment — the deep
tool, at the moment you want breadth. They now point here, and keep
`search` for reading a rule you already suspect. Written as a practice
rather than a prohibition (rule 165).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

tests/test_wide_net.py

@@ -0,0 +1,243 @@
+"""The wide net: many candidates, no bar, and a cost that stays bounded (#4103).
+
+WHY THIS EXISTS
+
+Milestone 416 step 5 moves "do not exclude potentially important data" off the
+push, where fifty candidates would be a wall nobody reads, and onto a pull,
+where they arrive only when a session asks for them.
+
+Three things have to hold, and each has a way of quietly failing:
+
+  1. **No bar.** The moment this tool serves is the one where the caller does
+     not trust a threshold to decide for them. A default floor creeping in
+     here would turn the wide net into the narrow one and nothing would look
+     wrong — the results would simply be fewer.
+  2. **A bounded cost.** The step was filed claiming fifty "costs nothing".
+     `_rule_hint_line` had already measured ~143 tokens for a line with its
+     trigger rendered, so fifty is ~7,000. The graduated shape (#3851) is what
+     keeps this affordable, and a change that renders every trigger in full
+     would pass every other test here.
+  3. **A pull, logged as one.** The push arms' near-miss distributions are the
+     evidence #4121 argues from. A pull mixed into them moves those numbers.
+"""
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from scribe.mcp.tools import wide_net
+from tests.helpers import FakeMCP, fake_rule, tool_doc
+
+
+def _hits(n=3, trigger="Running git push", **over):
+    """n (score, rule) pairs, descending, the shape the service returns."""
+    return [
+        (0.9 - i * 0.01,
+         fake_rule(id=i + 1, title=f"rule {i + 1}", when_to_apply=trigger, **over))
+        for i in range(n)
+    ]
+
+
+def _patched(hits, report=None):
+    """Patch the search and the telemetry sink; hand back the search mock."""
+    async def _search(*a, **kw):
+        if report is not None and "report" in kw and kw["report"] is not None:
+            kw["report"].update(report)
+        return hits
+    return patch.object(wide_net, "semantic_search_rules", AsyncMock(side_effect=_search))
+
+
+@pytest.mark.asyncio
+async def test_no_bar_reaches_the_search():
+    """THE POINT OF THE TOOL. A floor here would narrow the net silently."""
+    with _patched(_hits()) as search, \
+         patch.object(wide_net, "record_retrieval"), \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        await wide_net.what_might_apply("push to dev")
+
+    assert search.await_args.kwargs["threshold"] == 0.0
+    assert wide_net.THRESHOLD == 0.0
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("asked, expected", [(999, 50), (0, 1), (-5, 1), (25, 25)])
+async def test_the_limit_is_clamped_to_the_cap(asked, expected):
+    with _patched(_hits()) as search, \
+         patch.object(wide_net, "record_retrieval"), \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        await wide_net.what_might_apply("q", limit=asked)
+
+    assert search.await_args.kwargs["limit"] == expected
+
+
+@pytest.mark.asyncio
+async def test_the_head_carries_its_trigger_whole_and_the_tail_is_cut():
+    """The graduated shape (#3851), which is what makes fifty affordable.
+
+    A regression that rendered every trigger in full would satisfy every other
+    assertion in this file, so the cut is pinned on both sides: the head is
+    NOT marked truncated and the tail IS.
+    """
+    long_trigger = " ".join(["running a git command before pushing anything"] * 12)
+    with _patched(_hits(6, trigger=long_trigger)), \
+         patch.object(wide_net, "record_retrieval"), \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        out = await wide_net.what_might_apply("q", detail=2)
+
+    head, tail = out["candidates"][:2], out["candidates"][2:]
+    assert all(c["truncated"] is False for c in head)
+    assert all(c["when_to_apply"] == " ".join(long_trigger.split()) for c in head)
+    assert all(c["truncated"] is True for c in tail)
+    assert all(len(c["when_to_apply"]) <= wide_net._TEASER_CHARS + 1 for c in tail)
+    assert out["detailed"] == 2
+
+
+@pytest.mark.asyncio
+async def test_a_short_trigger_is_never_marked_truncated():
+    """`truncated` is a claim about this row, not about its rank. A tail row
+    whose trigger already fits must not claim a cut that did not happen."""
+    with _patched(_hits(4, trigger="Running git push")), \
+         patch.object(wide_net, "record_retrieval"), \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        out = await wide_net.what_might_apply("q", detail=1)
+
+    assert all(c["truncated"] is False for c in out["candidates"])
+
+
+def test_the_cut_breaks_on_a_word_and_says_it_was_cut():
+    """#4036's lesson, borrowed: a raw slice ends mid-word and reads as the
+    whole thing."""
+    text, cut = wide_net._teaser("alpha beta gamma delta " * 40)
+    assert cut is True
+    assert text.endswith("…")
+    # Broke on a word, so no partial token sits before the marker.
+    assert not text.removesuffix("…").rstrip().endswith(("alph", "bet", "gam"))
+
+
+def test_one_unbroken_word_still_yields_text_rather_than_a_bare_marker():
+    """`textwrap.shorten` returns just "…" here, which would render a teaser
+    carrying no information at all."""
+    text, cut = wide_net._teaser("x" * 500)
+    assert cut is True and text != "…"
+    assert len(text) == wide_net._TEASER_CHARS
+
+
+@pytest.mark.asyncio
+async def test_every_row_says_its_force_and_its_scope():
+    """`kind` is never inferred — a rule must be followed, a preference guides
+    — and `scope` says how far a reader should generalise from it."""
+    hits = [
+        (0.8, fake_rule(id=1, kind="rule", project_id=None)),
+        (0.7, fake_rule(id=2, kind="preference", project_id=44)),
+    ]
+    with _patched(hits), patch.object(wide_net, "record_retrieval"), \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        out = await wide_net.what_might_apply("q")
+
+    assert [c["kind"] for c in out["candidates"]] == ["rule", "preference"]
+    assert [c["scope"] for c in out["candidates"]] == ["global", "project"]
+
+
+# ── telemetry: a pull, and never mistaken for a push ────────────────────────
+
+@pytest.mark.asyncio
+async def test_the_call_is_logged_under_its_own_pull_source():
+    with _patched(_hits()), \
+         patch.object(wide_net, "record_retrieval") as rec, \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        await wide_net.what_might_apply("push to dev")
+
+    kw = rec.call_args.kwargs
+    assert kw["source"] == wide_net.SOURCE == "wide_net"
+    assert kw["threshold"] == 0.0
+
+
+def test_the_wide_net_is_not_one_of_the_tunable_push_surfaces():
+    """THE GUARD that keeps #4121's evidence clean (rule 167).
+
+    The registry holds the PUSH arms — the ones with a floor and a budget the
+    model tunes. This source must not appear there: a pull folded into those
+    rows would move the near-miss distributions that step is arguing from, and
+    would offer a floor to tune on a tool whose whole point is not having one.
+    """
+    from scribe.services.retrieval_surfaces import SURFACES
+
+    assert wide_net.SOURCE not in SURFACES
+
+
+def test_the_wide_net_is_not_ambient_either():
+    """Ambient means a delivery nobody chose. This one is chosen by definition
+    — somebody called the tool — so counting it as ambient would make a
+    deliberate ask read as a bulk hand-over."""
+    from scribe.services.note_usage import AMBIENT_SOURCES
+
+    assert wide_net.SOURCE not in AMBIENT_SOURCES
+
+
+@pytest.mark.asyncio
+async def test_a_search_that_never_ran_is_not_reported_as_a_decline():
+    """#3765: an empty query, a dead embedder and a failed query all return
+    nothing, and none of them is a ranker declining."""
+    with _patched([], report={"searched": False, "best_available_score": None}), \
+         patch.object(wide_net, "record_retrieval") as rec, \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        out = await wide_net.what_might_apply("")
+
+    assert rec.call_args.kwargs["searched"] is False
+    assert out["searched"] is False
+
+
+@pytest.mark.asyncio
+async def test_what_the_bar_turned_away_is_carried_through():
+    """There is no bar here, but `best_available` still answers "did the corpus
+    have anything at all" for a call that came back empty (#3670)."""
+    with _patched([], report={"searched": True, "best_available_score": 0.31,
+                              "best_available_id": 7}), \
+         patch.object(wide_net, "record_retrieval") as rec, \
+         patch.object(wide_net, "current_user_id", lambda: 1):
+        await wide_net.what_might_apply("q")
+
+    assert rec.call_args.kwargs["best_available"] == 0.31
+    assert rec.call_args.kwargs["best_available_id"] == 7
+
+
+# ── the contract a session actually reads ───────────────────────────────────
+
+def test_the_docstring_says_when_to_reach_for_it():
+    """The step's done-when, and the load-bearing half of this tool.
+
+    A wide net nobody knows to call is worth nothing, so the docstring has to
+    name the MOMENT, not just the parameters — including the one that prompted
+    it, where a session hands work back rather than finishing it.
+    """
+    doc = tool_doc("scribe.mcp.tools.wide_net", "what_might_apply").lower()
+
+    assert "consequential" in doc
+    assert "handing work back" in doc
+    # Says the tail is expected to be noise — otherwise the first caller reads
+    # a low-scoring list as the tool being broken.
+    assert "noise" in doc
+    # And names its own limit: a pull only helps if somebody asks.
+    assert "only helps if you ask" in doc
+
+
+def test_it_is_registered_and_readable_with_a_read_key():
+    from scribe.mcp.server import _READ_ONLY_TOOLS
+
+    mcp = FakeMCP()
+    wide_net.register(mcp)
+    assert mcp.names == ["what_might_apply"]
+    assert "what_might_apply" in _READ_ONLY_TOOLS
+
+
+def test_the_instruction_surfaces_point_at_it():
+    """Rule 119: the instruction surfaces ARE the specification for product
+    behaviour, so a tool the reflex never learns about is not shipped."""
+    import pathlib
+
+    from scribe.mcp import server
+
+    assert "what_might_apply" in server._INSTRUCTIONS
+
+    skill = (pathlib.Path(__file__).resolve().parents[1]
+             / "plugin" / "skills" / "using-scribe" / "SKILL.md").read_text()
+    assert "what_might_apply" in skill