feat(rules): a slot a preference cannot lose (#3894)

Milestone 399 step 4. A rule and a preference are not equally served by one
ranking, because their losses are not equal:

  - a RULE crowded out at the prompt boundary still fires at an act arm. A
    push reaches pre_tool_rule, a write reaches write_path_rule. The prompt
    hit is a preview of a second chance.
  - a PREFERENCE about how to answer has no second chance. The response IS
    the act, so crowded out there it is never delivered at all.

A straight ranking therefore favours the record whose loss is recoverable
over the one whose loss is total, and does it INVISIBLY: the rule that won is
a legitimate hit, the telemetry reads healthy, and the only symptom is a
preference that quietly never arrives. reuse_slot exists for the same shape
one corpus over (#2463).

`semantic_search_rules` gains a `kind` filter, so the slot's query can only
answer with what the slot is for. Verifying afterwards would be weaker — an
unfiltered search that happened to return a rule would spend the slot on it,
and that line would be indistinguishable from one that earned its place.

THE SLOT BUYS POSITION, NOT A LOWER BAR, matching reuse_slot. A weak
preference cannot buy it, so silence stays the default. The task asked for a
separate threshold; I did not add one, and the reason is that the worry
behind it — reading a miss rate as a fact about preferences — is answered by
`preference_slot` being its own logged source, where best_available_id names
which preference was refused. A knob added on a guess is a way to
misconfigure the surface; a bar moved on evidence is an argument. The
evidence arrives on its own now.

IT EXTENDS, IT NEVER DISPLACES — and here it parts from reuse_slot, which
evicts its menu's weakest hit. A displaced hit sits in prompt_rule's
retrieval_logs row while never being surfaced, so that source's two tables
stop agreeing and #3668's identity breaks for a reason nothing in the data
explains. Milestone #379 is what losing that identity costs: five steps
planned against two counters disagreeing, not a write path dropping rows. One
extra line in a rare case is the cheaper price.

It also runs BEFORE the bail-out. An empty general result is not proof no
preference qualifies: that search overfetches by distance then collapses, so
a preference ranked below the window is invisible to it while a kind-filtered
query finds it at once. Bailing first would make the slot dead in exactly the
corpus it exists for.

One existing assertion repinned from a bare call_count to a per-source
filter: the slot logs its own query on the same call, and a count would pin
the number of arms rather than the property — going red the next time one is
added, which is rule 167's false alarm about the thing it protects.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/services/embeddings.py

@@ -818,6 +818,7 @@ async def semantic_search_rules(
     limit: int = 5,
     threshold: float = _SIMILARITY_THRESHOLD,
     tier: str | None = None,
+    kind: str | None = None,
     report: dict | None = None,
 ) -> list[tuple[float, "Rule"]]:
     """Return up to *limit* (score, rule) pairs most relevant to *query*.
@@ -855,6 +856,16 @@ async def semantic_search_rules(
     Pass a tier when a caller genuinely wants one class — a listing, an audit,
     a UI that renders the tiers apart. Not to approximate relevance.
 
+    `kind` narrows to `rule` or `preference`, and NONE is likewise the ordinary
+    case: a caller asking "what governs this" wants both, because the reader
+    needs to know what binds AND how the operator wants it done. The one place
+    it is passed is a RESERVED SLOT — a query that may only return a
+    preference, so the slot cannot be spent on something else. That is the
+    same reason `note_type` exists on the sibling search, and the same failure
+    it prevents: a slot silently filled by the wrong kind is worse than no
+    slot, because the line is indistinguishable from one that earned its place
+    on score.
+
     Collapses to best-chunk-per-rule like the note search, so a long rule split
     across chunks competes once rather than crowding the results with itself.
 
@@ -895,6 +906,7 @@ async def semantic_search_rules(
                         Project.user_id == user_id,
                     ),
                     *( [Rule.tier == tier] if tier else [] ),
+                    *( [Rule.kind == kind] if kind else [] ),
                 )
                 # Overfetch so collapsing chunks to their best row still fills
                 # the page — the same reason the note search overfetches.

src/scribe/services/plugin_context.py

@@ -678,6 +678,109 @@ async def build_autoinject_hint(
     return {"context": "\n".join(lines), "note_ids": note_ids, "config": cfg}
 
 
+async def _reserve_slot_for_preference(
+    user_id: int,
+    query: str,
+    hits: list,
+    *,
+    threshold: float,
+    project_id: int,
+    already: set[int],
+) -> tuple[list, int | None]:
+    """Guarantee a preference one slot, if one clears the bar (#3894).
+
+    THE ASYMMETRY THIS EXISTS FOR. A rule and a preference are not equally
+    served by a shared score contest, because their losses are not equal:
+
+      - a RULE crowded out here can still fire at the act arm. A `git push`
+        reaches `pre_tool_rule`, a file write reaches `write_path_rule`. The
+        prompt hit is a preview of a second chance.
+      - a PREFERENCE about how to answer has no second chance. There is no
+        later act — the response IS the act — so crowded out here it is never
+        delivered at all.
+
+    A straight ranking therefore favours the record whose loss is recoverable
+    over the one whose loss is total, and it does so INVISIBLY: the rule that
+    won is a legitimate hit, the telemetry looks healthy, and the only symptom
+    is a preference that quietly never arrives. `reuse_slot` exists for the
+    same shape one corpus over (#2463), where snippets kept losing to project
+    records that merely resembled the query.
+
+    THE SLOT BUYS POSITION, NOT A LOWER BAR — same as `reuse_slot`, which also
+    reserves at `cfg["threshold"]`. A weak preference cannot buy the slot, so
+    silence stays the default and the reserved line is never worse than the
+    ones it sits beside. If `preference_slot` later shows a stream of
+    near-misses, `best_available_id` (#3807) names which preference was
+    refused and a separate bar becomes an argument with evidence behind it
+    rather than a knob added on a guess.
+
+    LEDGER REPEATS STILL COUNT AS REPRESENTED. A preference already on the
+    session's ledger occupies the slot rather than being skipped for a fresh
+    one: it is still rendered (#3750), just with the tail that says so, and a
+    preference is the kind of record where being reminded is the point.
+
+    Returns the possibly-extended hit list, and the id the slot spent — the
+    caller needs that to keep each source's surfaced set matching its own log
+    row (#3668), since the slot logs under its own name.
+    """
+    if any(rule.kind == "preference" for _s, rule in hits):
+        return hits, None
+
+    _t0 = time.perf_counter()
+    _rep: dict = {}
+    # KIND-FILTERED, so the query can only answer with what the slot is for.
+    # Verifying the kind afterwards would be weaker: an unfiltered search that
+    # happened to return a rule would spend the slot on it, and the line would
+    # be indistinguishable from one that earned its place.
+    found = await semantic_search_rules(
+        user_id, query, limit=1, threshold=threshold,
+        kind="preference", report=_rep,
+    )
+    fresh = [(s, r) for s, r in found if r.id not in already]
+    # ITS OWN SOURCE, and both sides of the trade logged. #2463's own finding
+    # is the warning rather than the precedent here: the hit that slot pushed
+    # OUT was in retrieval_logs while the query that pushed it out was not, so
+    # the slot could never be judged against what it displaced. `results` is
+    # fresh-only, matching what gets recorded as surfaced below (#3752/#3668).
+    record_retrieval(
+        user_id=user_id, source="preference_slot", query=query,
+        threshold=threshold, limit=1, project_id=project_id,
+        is_task=None, results=fresh,
+        best_available=_rep.get("best_available_score"),
+        best_available_id=_rep.get("best_available_id"),
+        searched=bool(_rep.get("searched", True)),
+        suppressed=len(found) - len(fresh),
+        duration_ms=(time.perf_counter() - _t0) * 1000.0,
+    )
+    seen = {rule.id for _s, rule in hits}
+    slot = [(s, r) for s, r in found
+            if r.kind == "preference" and r.id not in seen][:1]
+    if not slot:
+        return hits, None
+
+    slot_id = int(slot[0][1].id)
+    if slot_id not in already:
+        record_rule_surfaced(
+            user_id=user_id, rule_ids=[slot_id], source="preference_slot",
+        )
+    # IT EXTENDS, IT NEVER DISPLACES — and here it parts company with
+    # `reuse_slot`, which evicts its menu's weakest hit. The reason is the
+    # ledger rather than taste. A displaced hit was RETURNED by the general
+    # search and is sitting in that call's `retrieval_logs` row, but would not
+    # have been shown — so `prompt_rule`'s surfaced set would stop matching
+    # its own log row, and #3668's identity would break for a reason nothing
+    # in the data explains. That identity is the cheapest true statement
+    # available about this pair of tables, and milestone #379 is what it costs
+    # to lose it: five steps planned against a gap that was two counters
+    # disagreeing, not a write path dropping rows.
+    #
+    # The price is one extra line, only when the general search already filled
+    # the limit AND a preference cleared the bar without placing. Cheap, and
+    # it buys a surface whose two tables can always be checked against each
+    # other.
+    return hits + slot, slot_id
+
+
 async def build_prompt_rule_hint(
     user_id: int,
     query: str,
@@ -758,6 +861,18 @@ async def build_prompt_rule_hint(
             searched=bool(_rep.get("searched", True)),
             suppressed=len(hits) - len(fresh),
         )
+        # THE RESERVED SLOT RUNS BEFORE THE BAIL-OUT, and that ordering is
+        # load-bearing rather than tidy. An empty general result is not proof
+        # that no preference qualifies: the general search overfetches by
+        # distance and then collapses, so a preference ranked below that
+        # window is invisible to it while a kind-filtered query finds it at
+        # once. Bailing first would make the slot dead in exactly the corpus
+        # it exists for — one where rules outnumber preferences.
+        hits, slot_id = await _reserve_slot_for_preference(
+            user_id, q, hits, threshold=threshold,
+            project_id=project_id, already=already,
+        )
+
         # `hits`, not `fresh` (#3750): a call whose only hit is a repeat still
         # has something to say, it just says it differently.
         if not hits:
@@ -770,6 +885,11 @@ async def build_prompt_rule_hint(
         # FRESH-ONLY (#3752). A reference is a rendering decision, not a
         # retrieval outcome, and counting one here would inflate the
         # denominator pull_through is read from.
+        #
+        # `fresh` is the PRE-SLOT list on purpose: it is exactly what this
+        # call's own `retrieval_logs` row recorded, so the two stay equal
+        # (#3668). The slot's hit is surfaced under `preference_slot` by the
+        # helper, against that source's own row.
         rule_ids = [rule.id for _score, rule in fresh]
 
         # RANKED, not ambient: this arm chose what it showed. The name is also

src/scribe/services/rule_usage.py

@@ -97,7 +97,15 @@ logger = logging.getLogger(__name__)
 # surfacing is a claim ("this rule may apply to what you are doing") that a pull
 # can confirm or refute, while an ambient one is a delivery nobody decided on.
 # Add a source here only when a ranker picked it.
-RANKED_SOURCES = ("write_path_rule", "pre_tool_rule", "prompt_rule")
+RANKED_SOURCES = (
+    "write_path_rule", "pre_tool_rule", "prompt_rule",
+    # A reserved slot is a ranker's choice twice over — it ran a query AND
+    # decided a kind was worth guaranteeing a place. Left out, its line would
+    # be counted as bulk delivery and drop out of the denominator, so the one
+    # surface built because a record class kept losing would be the one whose
+    # hits nobody could confirm.
+    "preference_slot",
+)
 
 
 def is_ambient(source: str) -> bool:

tests/test_rule_usage_wiring.py

@@ -554,12 +554,17 @@ async def test_the_prompt_arm_logs_the_calls_that_found_nothing():
     out = await _run_prompt_arm([], MagicMock(), retrieval_log=log)
 
     assert out["context"] == ""
-    assert log.call_count == 1, (
+    # BY SOURCE, not by count. The reserved slot (#3894) logs its own query on
+    # the same call, so a bare call_count would pin the number of arms rather
+    # than the property — and would go red the next time one is added, which
+    # is rule 167's false alarm about the very thing being protected.
+    general = [c for c in log.call_args_list
+               if c.kwargs.get("source") == "prompt_rule"]
+    assert len(general) == 1, (
         "the prompt arm returned early without logging a call that found "
         "nothing — the only evidence its inherited threshold is too high"
     )
-    assert log.call_args.kwargs["source"] == "prompt_rule"
+    assert general[0].kwargs["results"] == []
-    assert log.call_args.kwargs["results"] == []
 
 
 @pytest.mark.asyncio
@@ -1335,3 +1340,217 @@ async def test_a_reference_is_rendered_but_not_counted(source, run):
         "whole context will not do: the write-path arm fills it from four "
         "other sources."
     )
+
+
+# ── the reserved preference slot (#3894) ────────────────────────────────
+#
+# A rule and a preference are not equally served by one ranking, because their
+# losses are not equal. A rule crowded out at the prompt boundary still fires
+# at an act arm — a push reaches pre_tool_rule, a write reaches
+# write_path_rule. A preference about how to ANSWER has no later act: the
+# response is the act, so crowded out here it is never delivered at all.
+#
+# The failure is invisible without this slot. The rule that won is a
+# legitimate hit, the telemetry reads healthy, and the only symptom is a
+# preference that quietly never arrives — which is `reuse_slot`'s shape one
+# corpus over (#2463), where snippets kept losing to project records that
+# merely resembled the query.
+
+
+def _search_by_kind(general, preference):
+    """Stand in for the two calls the arm makes against one corpus.
+
+    The arm searches twice: once across every kind, once filtered to
+    preferences for the slot. A single return value cannot tell those apart,
+    and a test that could not tell them apart would pass against an arm that
+    never filtered at all — which is the one thing making the slot a slot.
+    """
+    async def _search(*_args, **kwargs):
+        return preference if kwargs.get("kind") == "preference" else general
+    return AsyncMock(side_effect=_search)
+
+
+async def _run_slot(general, preference, recorder=None, retrieval_log=None, **kwargs):
+    from scribe.services import plugin_context as pc
+    rec = recorder or MagicMock()
+    with ExitStack() as stack:
+        stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
+        stack.enter_context(patch.object(
+            pc, "semantic_search_rules", _search_by_kind(general, preference)))
+        stack.enter_context(patch.object(
+            pc, "record_retrieval", retrieval_log or MagicMock()))
+        stack.enter_context(patch.object(pc, "record_rule_surfaced", rec))
+        out = await pc.build_prompt_rule_hint(1, "please merge to main", **kwargs)
+    return out, rec
+
+
+_PREF_HIT = [(0.74, fake_rule(
+    id=140, kind="preference", title="Let each action land before the next",
+    when_to_apply="before starting an action while a previous one is settling",
+))]
+_RULES_FILLING_THE_LIMIT = [
+    (0.81, fake_rule(id=2, title="`main` — never without explicit request")),
+    (0.79, fake_rule(id=1, title="`dev` is home")),
+    (0.77, fake_rule(id=153, title="Merge dev→main with a plain merge commit")),
+]
+
+
+@pytest.mark.asyncio
+async def test_a_preference_that_lost_the_ranking_still_gets_a_line():
+    """The whole point. Three rules fill the limit; the preference places
+    fourth on score and would never be seen without the slot."""
+    out, _rec = await _run_slot(_RULES_FILLING_THE_LIMIT, _PREF_HIT)
+
+    assert "get_rule(140)" in out["context"], (
+        "a preference cleared the bar, placed behind the rules, and was "
+        "dropped — which is the outcome with no symptom: the rules that won "
+        "are legitimate hits and nothing in the telemetry looks wrong"
+    )
+    assert "Preference that may apply" in out["context"]
+
+
+@pytest.mark.asyncio
+async def test_the_slot_is_not_spent_when_a_preference_already_placed():
+    """A guaranteed slot is a floor, not a quota. A preference that earned its
+    place on score does not entitle the corpus to a second one."""
+    general = [(0.81, fake_rule(id=140, kind="preference", title="Let each action land"))]
+    search_log = MagicMock()
+    out, _rec = await _run_slot(general, _PREF_HIT, retrieval_log=search_log)
+
+    sources = [c.kwargs.get("source") for c in search_log.call_args_list]
+    assert "preference_slot" not in sources, (
+        "the slot ran while a preference had already placed — a second "
+        "reserved line for a kind already represented is noise the general "
+        "ranking had already decided against"
+    )
+    assert out["context"].count("Preference that may apply") == 1
+
+
+@pytest.mark.asyncio
+async def test_the_slot_query_can_only_answer_with_a_preference():
+    """Filtered at the QUERY, not verified afterwards.
+
+    An unfiltered search that happened to return a rule would spend the slot
+    on it, and that line would be indistinguishable from one that earned its
+    place on score — a slot silently spent on the wrong kind is worse than no
+    slot at all.
+    """
+    from scribe.services import plugin_context as pc
+    search = _search_by_kind(_RULES_FILLING_THE_LIMIT, _PREF_HIT)
+    with ExitStack() as stack:
+        stack.enter_context(patch.object(pc, "get_setting", AsyncMock(return_value="0.6")))
+        stack.enter_context(patch.object(pc, "semantic_search_rules", search))
+        stack.enter_context(patch.object(pc, "record_retrieval", MagicMock()))
+        stack.enter_context(patch.object(pc, "record_rule_surfaced", MagicMock()))
+        await pc.build_prompt_rule_hint(1, "please merge to main")
+
+    kinds = [c.kwargs.get("kind") for c in search.await_args_list]
+    assert "preference" in kinds, "the slot searched without filtering by kind"
+    assert kinds.count("preference") == 1, "the slot searched more than once"
+
+
+@pytest.mark.asyncio
+async def test_the_slot_runs_even_when_the_general_search_found_nothing():
+    """The ordering the arm gets wrong by default.
+
+    An empty general result is not proof no preference qualifies: that search
+    overfetches by distance and then collapses, so a preference ranked below
+    the window is invisible to it while a kind-filtered query finds it at
+    once. Bailing out first would make the slot dead in exactly the corpus it
+    exists for — one where rules outnumber preferences.
+    """
+    out, _rec = await _run_slot([], _PREF_HIT)
+    assert "get_rule(140)" in out["context"], (
+        "the arm returned early on an empty general result and never asked "
+        "for a preference"
+    )
+
+
+@pytest.mark.asyncio
+async def test_the_slot_extends_rather_than_displacing():
+    """Nothing the general search returned goes un-shown.
+
+    `reuse_slot` evicts its menu's weakest hit; this one does not, and the
+    reason is the ledger. A displaced hit sits in `prompt_rule`'s
+    retrieval_logs row while never being surfaced, so that source's two
+    tables stop agreeing — and #3668's identity is the cheapest true
+    statement available about this pair. Milestone #379 is what losing it
+    costs: five steps planned against two counters disagreeing.
+    """
+    log, rec = MagicMock(), MagicMock()
+    out, _ = await _run_slot(
+        _RULES_FILLING_THE_LIMIT, _PREF_HIT, recorder=rec, retrieval_log=log,
+    )
+
+    for rule_id in (2, 1, 153):
+        assert f"get_rule({rule_id})" in out["context"], (
+            f"rule {rule_id} was returned and logged, then pushed out by the "
+            f"slot — surfaced and logged now disagree for prompt_rule"
+        )
+
+    logged = [
+        r.id for c in log.call_args_list if c.kwargs.get("source") == "prompt_rule"
+        for _s, r in c.kwargs["results"]
+    ]
+    surfaced = [
+        rid for c in rec.call_args_list if c.kwargs.get("source") == "prompt_rule"
+        for rid in c.kwargs["rule_ids"]
+    ]
+    assert logged == surfaced, (
+        f"prompt_rule logged {logged} and surfaced {surfaced} — the identity "
+        f"#3668 pins, broken by the slot rather than by a write path"
+    )
+
+
+@pytest.mark.asyncio
+async def test_the_slot_accounts_for_itself_under_its_own_source():
+    """Both sides of the trade, logged.
+
+    #2463's own finding is the warning rather than the precedent: the hit
+    that slot pushed OUT was in retrieval_logs while the query that pushed it
+    out was not, so the slot could never be judged against what it displaced.
+    This one logs its query AND records its surfacing, under a source of its
+    own, so it can be evaluated separately from the ranking it bypassed.
+    """
+    log, rec = MagicMock(), MagicMock()
+    await _run_slot(_RULES_FILLING_THE_LIMIT, _PREF_HIT,
+                    recorder=rec, retrieval_log=log)
+
+    slot_logs = [c for c in log.call_args_list
+                 if c.kwargs.get("source") == "preference_slot"]
+    assert len(slot_logs) == 1, "the slot ran without logging its own query"
+    assert [r.id for _s, r in slot_logs[0].kwargs["results"]] == [140]
+
+    slot_surfacings = [c for c in rec.call_args_list
+                       if c.kwargs.get("source") == "preference_slot"]
+    assert len(slot_surfacings) == 1
+    assert slot_surfacings[0].kwargs["rule_ids"] == [140]
+
+
+@pytest.mark.asyncio
+async def test_a_preference_on_the_ledger_keeps_the_slot_and_is_not_recounted():
+    """Repeats hold the slot; they just do not count twice.
+
+    A preference is the kind of record where being reminded is the point, so
+    one already on the ledger occupies the slot rather than being skipped for
+    a fresh one — rendered with the repeat tail (#3750). What it must not do
+    is register a second surfacing, which would count one delivery twice in
+    the denominator pull-through is read from (#3752).
+    """
+    log, rec = MagicMock(), MagicMock()
+    out, _ = await _run_slot(
+        _RULES_FILLING_THE_LIMIT, _PREF_HIT,
+        recorder=rec, retrieval_log=log, exclude_rule_ids=[140],
+    )
+
+    assert "get_rule(140)" in out["context"]
+    assert _SEEN_TAIL in out["context"], "the repeat was rendered as a first surfacing"
+    assert not [c for c in rec.call_args_list
+                if c.kwargs.get("source") == "preference_slot"], (
+        "a preference the session had already been shown was counted as a "
+        "fresh surfacing"
+    )
+    assert 140 not in out["rule_ids"], (
+        "a repeat was written back to the hook's ledger, which would keep "
+        "pushing its stamp forward so it never aged out (#3751)"
+    )