feat(retrieval): the wide net becomes a pull — fifty candidates, no bar (#4103)

Milestone 416 step 5. The operator's compromise — "if you're worried
about excluding potentially important data let limit it to 50 entries"
— moved to the surface where it is safe. Fifty in the push would be
milestone 394 with extra steps; fifty in a pull crowds nothing out.

`what_might_apply(query)` returns ranked rule candidates with NO
threshold. The moment it serves is the one where the caller does not
trust a bar to decide for them, so it does not have one — every row
carries its score and the reader judges.

WHY IT IS NOT A BIGGER `limit` ON `search`

`_search_rules` returns statement, why and how_to_apply in full, on the
stated reasoning that a caller who went looking deserves the whole
record. That is the DEEP pull and should stay that way. This is the
SHALLOW one — many candidates, each just enough to decide whether to
open it. Opposite trade-offs, so it is a second tool.

THE PREMISE THE STEP GOT WRONG

The task said fifty "costs nothing". `_rule_hint_line` had already
measured otherwise: ~143 tokens per line once the trigger is rendered,
and #3855 tripled trigger lengths across the corpus. Fifty is ~7,000
tokens — cheap next to an arm firing before every Bash call, but not
free, and a tool promising a free wide net gets reached for casually
and then regretted.

So it reuses the graduated shape #3851 measured for the push: the top
few carry their trigger whole, the rest carry a cut of it. TRUNCATED,
never dropped — the trigger is what lets a reader judge without
opening, and a teaser without one is just an id. The cut borrows
`_goal_line`'s technique including the fallback that matters (#4036):
`textwrap.shorten` returns a bare "…" for one unbroken word.

TELEMETRY

Logged under its own pull source, asserted absent from both the tunable
push registry and AMBIENT_SOURCES. That guard is load-bearing right
now: the push arms' near-miss distributions are the evidence #4121
argues from, and a pull folded into them would move those numbers.

INSTRUCTION SURFACES

The using-scribe reflex and the MCP instructions both pointed at
`search(content_type="rule")` for the consequential moment — the deep
tool, at the moment you want breadth. They now point here, and keep
`search` for reading a rule you already suspect. Written as a practice
rather than a prohibition (rule 165).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/mcp/server.py

@@ -44,8 +44,11 @@ client reads Agent Skills) and in each tool's description. The index:
   system. An `inception` key: ask what it inherits, then
   decide_project_inception.
 - RULES: nothing preloads; a rule arrives when your work matches it. Before a
-  consequential act, search(content_type="rule"). Silence means
-  nothing matched, not none. Rules bind; preferences guide.
+  consequential act — or before handing work back unsure you may finish it —
+  what_might_apply("what you are about to do"): the wide net, fifty ranked
+  candidates, no bar. search(content_type="rule") reads one you already
+  suspect. Silence means nothing matched, not none. Rules bind; preferences
+  guide.
 - RECALL: search before acting, scoped with the active project_id.
 - RECORD: create_task; a fix is kind="issue". add_task_log as you go; status
   in_progress on start, done on finish. Tag system_ids as you write.
@@ -137,6 +140,11 @@ _READ_ONLY_TOOLS = frozenset({
     # usual for these two: a session that cannot see the bar in force, or the
     # reason it was last moved, is a session that will move it again blind.
     "retrieval_surfaces", "retrieval_tuning_history",
+    # The wide net (#4103) — ranked rule candidates with no bar, for the
+    # moment before a consequential act. A pure read, and one a read key needs
+    # most: it is the surface the "ask before acting" reflex calls, and a key
+    # that could not reach it would be denied exactly the check it should run.
+    "what_might_apply",
 })
 
 # Every tool that WRITES, by name. Nothing reads this set at runtime — a tool

src/scribe/mcp/tools/__init__.py

@@ -6,6 +6,7 @@ from `mcp.server.build_mcp_server`.
 """
 from scribe.mcp.tools import (
     design_systems, milestones, notes, processes, projects, recent, repos, retrieval_tuning,
+    wide_net,
     rulebooks, search, shapes, snippets, systems, tags, tasks, trash,
 )
 
@@ -14,6 +15,7 @@ def register_all(mcp) -> None:
     """Register every tool module's tools on the given FastMCP instance."""
     search.register(mcp)
     retrieval_tuning.register(mcp)
+    wide_net.register(mcp)
     notes.register(mcp)
     tasks.register(mcp)
     projects.register(mcp)

src/scribe/mcp/tools/wide_net.py

@@ -0,0 +1,218 @@
+"""The wide net — every candidate that might govern what you are about to do (#4103).
+
+WHY THIS EXISTS
+
+Milestone 416 step 5, from the operator's compromise:
+
+    "if you're worried about excluding potentially important data let limit it
+    to 50 entries or something like that"
+
+Fifty in the PUSH would be milestone 394 with extra steps — a wall the reader
+skims, with the governing record indistinguishable from forty-nine others.
+Fifty in a PULL is a different object: it arrives only when a session asks, it
+crowds nothing out, and it is the honest home for "do not exclude potentially
+important data".
+
+WHY IT IS NOT `search(content_type="rule")`
+
+That tool is the DEEP pull and should stay that way — `_search_rules` returns
+`statement`, `why` and `how_to_apply` in full, on the reasoning that a caller
+who went looking deserves the whole record rather than a summary to re-fetch.
+Six results already run to thousands of tokens.
+
+This is the SHALLOW pull: many candidates, each just enough to decide whether
+to open it. Opposite trade-off, so it is a second tool rather than a bigger
+`limit` on the first.
+
+WHY THE TAIL IS TRUNCATED RATHER THAN FULL
+
+Because the step's premise needed correcting. The task said fifty "costs
+nothing", and `_rule_hint_line` had already measured otherwise: a line runs
+~143 tokens once its trigger is rendered, and #3855 tripled trigger lengths
+across the corpus. Fifty of those is ~7,000 tokens — cheap next to an arm that
+fires before every Bash call, but not free, and a tool that promises a free
+wide net gets reached for casually and then regretted.
+
+So this reuses the graduated shape #3851 measured for the push: the top few
+carry their trigger in full, the rest carry a cut of it. TRUNCATED, never
+dropped — the trigger is what lets a reader judge relevance without opening
+the record, and a teaser without one is just an id.
+"""
+from __future__ import annotations
+
+import textwrap
+import time
+
+from scribe.mcp._context import current_user_id
+from scribe.services.embeddings import semantic_search_rules
+from scribe.services.retrieval_telemetry import record_retrieval
+
+# The telemetry `source`, and a PULL — never added to AMBIENT_SOURCES (those
+# are deliveries nobody chose) and never confused with a push arm. Keeping it
+# separate is load-bearing right now: the push arms' near-miss distributions
+# are the evidence #4121 rests on, and a pull mixed into them would move the
+# very numbers that step is arguing from.
+SOURCE = "wide_net"
+
+# No bar. That is the point of the tool rather than an oversight — the caller
+# asked for the wide net precisely because they do not trust a bar to decide
+# for them here. Every row carries its score, so the reader sees where the
+# ranking falls off and judges it themselves.
+THRESHOLD = 0.0
+
+MAX_LIMIT = 50
+DEFAULT_LIMIT = 25
+
+# How many candidates carry their trigger in full before the rest are cut.
+DEFAULT_DETAIL = 5
+
+# The cut length for the tail. Long enough to carry the first clause of a
+# trigger — which is where these state the act they are about — short enough
+# that forty-five of them stay affordable.
+_TEASER_CHARS = 140
+
+
+def _teaser(text: str) -> tuple[str, bool]:
+    """Shorten at a word break with a visible cut. Returns (text, was_cut).
+
+    The technique is `plugin_context._goal_line`'s, and the fallback is the
+    part worth copying: `textwrap.shorten` returns a bare "…" when the string
+    is one unbroken word longer than the cap, and a raw slice ends mid-word
+    claiming to be the whole thing (#4036). Kept local rather than shared
+    because the two callers wrap it in different sentences; if a third appears,
+    that is the moment to extract it rather than now.
+    """
+    flat = " ".join((text or "").split())
+    if len(flat) <= _TEASER_CHARS:
+        return flat, False
+    short = textwrap.shorten(flat, width=_TEASER_CHARS, placeholder="…")
+    if short == "…":
+        short = flat[: _TEASER_CHARS - 1] + "…"
+    return short, True
+
+
+async def what_might_apply(
+    query: str,
+    limit: int = DEFAULT_LIMIT,
+    detail: int = DEFAULT_DETAIL,
+    kind: str = "",
+    project_id: int = 0,
+) -> dict:
+    """Every rule that might bear on what you are about to do, ranked, with no bar.
+
+    REACH FOR THIS BEFORE A CONSEQUENTIAL OR IRREVERSIBLE ACT — a push, a
+    merge, a delete, a deploy, anything outward-facing — and before handing
+    work back because you are unsure whether you are allowed to finish it.
+
+    It is the tool the "ask before a consequential act" reflex should call.
+    A rule reaches a session by retrieval, and the arms that push rules at you
+    have a small budget: they deliver the few highest-scoring candidates and
+    say nothing about what sat just underneath. That is right for an arm that
+    fires before every command and wrong for the one moment you actually want
+    to be sure. This is that moment's tool.
+
+    WHAT IT RETURNS, AND WHY THE TAIL LOOKS LIKE NOISE
+
+    There is no threshold. You get the `limit` nearest candidates whatever they
+    score, ordered, each with its score — so the tail IS expected to be
+    irrelevant, and that is the design. You are not reading the list for its
+    average quality; you are reading it for the one record you would otherwise
+    have missed. Scan the triggers, open what looks live with `get_rule(id)`,
+    and ignore the rest.
+
+    The first few carry their trigger in full; the rest carry a cut of it,
+    marked `truncated`. A cut trigger is still enough to decide whether to
+    open the record, which is the whole job of a teaser.
+
+    `kind` is on every row and is never something to infer: a **rule** must be
+    followed, a **preference** records how the operator wants work done.
+    Missing a rule is a mistake; missing a preference costs consistency.
+
+    A NOTE ON WHAT THIS CANNOT DO. It is a pull, so it only helps if you ask.
+    The failure it was built for — a session withholding a routine action
+    because the rule permitting it never arrived — produces no tool call of its
+    own, so nothing will prompt you. Asking is the habit; this is where to put
+    it.
+
+    Args:
+        query: what you are about to do, in the words you would use to
+            describe it — "push to dev after committing", "delete the staging
+            database", "merge dev to main". A command string works; a sentence
+            usually works better, because triggers are written as prose.
+        limit: how many candidates, default 25, capped at 50.
+        detail: how many carry their trigger in FULL before the rest are cut,
+            default 5.
+        kind: "rule" or "preference" to restrict; omit for both.
+        project_id: scope to one project — its own rules plus every global
+            one. Omit to ask the whole rulebook.
+    """
+    uid = current_user_id()
+    limit = max(1, min(int(limit), MAX_LIMIT))
+    detail = max(0, min(int(detail), limit))
+
+    report: dict = {}
+    started = time.perf_counter()
+    if project_id:
+        raw = await semantic_search_rules(
+            uid, query, limit=limit, threshold=THRESHOLD, kind=kind or None,
+            report=report, project_id=project_id,
+        )
+    else:
+        raw = await semantic_search_rules(
+            uid, query, limit=limit, threshold=THRESHOLD, kind=kind or None,
+            report=report, everywhere=True,
+        )
+    duration_ms = (time.perf_counter() - started) * 1000
+
+    candidates = []
+    for rank, (score, rule) in enumerate(raw):
+        full = rank < detail
+        trigger, was_cut = (
+            (" ".join((rule.when_to_apply or "").split()), False)
+            if full else _teaser(rule.when_to_apply or "")
+        )
+        candidates.append({
+            "id": rule.id,
+            "title": rule.title,
+            # Force, not topic. See the docstring — this is never inferred.
+            "kind": rule.kind,
+            # A rule in a rulebook topic is global; one on a project binds
+            # there alone (milestone 414). Which it is changes how far a
+            # reader should generalise from it.
+            "scope": "project" if rule.project_id else "global",
+            "when_to_apply": trigger,
+            "truncated": was_cut,
+            "score": round(float(score), 4),
+        })
+
+    # Logged as a pull, with `searched` honoured: a search that never ran must
+    # not be recorded as a ranker declining (#3765).
+    record_retrieval(
+        user_id=uid,
+        source=SOURCE,
+        query=query,
+        threshold=THRESHOLD,
+        limit=limit,
+        project_id=project_id or None,
+        is_task=None,
+        results=raw,
+        duration_ms=duration_ms,
+        best_available=report.get("best_available_score"),
+        best_available_id=report.get("best_available_id"),
+        searched=report.get("searched", True),
+    )
+
+    return {
+        "candidates": candidates,
+        "returned": len(candidates),
+        "detailed": min(detail, len(candidates)),
+        # Present so a caller can tell "the corpus offered nothing" from "the
+        # search never ran" — an empty query, an unavailable embedder and a
+        # failed query all return zero rows and mean different things (#3670).
+        "searched": bool(report.get("searched", True)),
+        "open_with": "get_rule(id)",
+    }
+
+
+def register(mcp) -> None:
+    mcp.tool(name="what_might_apply")(what_might_apply)