FabledScribe/src/scribe/services/retrieval_surfaces.py

"""One registry of the retrieval surfaces, and the two numbers each one has (#4102).

WHY THIS EXISTS

Six push arms each carried their own loose copy of the same shape: a settings
key, a default, and a limit that was usually a module constant nobody could
change. The read-and-clamp was written out separately in `plugin_context` (twice
over, for auto-inject and the write path), in three rule arms, and again as
`reply_preferences._threshold`. That was survivable while the numbers were
shipped constants an operator occasionally edited.

It stops being survivable once the numbers are meant to MOVE. The operator's
decision for this step:

    "the floor should be chosen and adjusted by the model using it. we've come
    back to something either fails or has to be looked at by the user we need a
    model consistent surface for the adjustment of these floor values. the user
    should be able to touch it but the model should be the thing handling it 9
    times out of 10."

A tuning surface cannot be "model consistent" if every arm spells its own
configuration differently. So the arms stop owning their numbers and read them
from here instead, and the tuning tool, the routes and the Settings UI all
enumerate THIS table rather than hard-coding six special cases.

WHAT A FLOOR IS NOW, AND WHAT IT IS NOT

It is not a relevance judgement. Relevance is decided by the reader, which is
the only participant that can read a trigger against a situation — that is the
milestone's whole argument, and the injected line has always said so out loud
("read it before deciding it does not apply").

A floor answers the cheaper question: *is this worth ranking at all*. `k` is
what binds, and `k` is a BUDGET — how much of this surface's attention a
candidate list may spend. An arm that fires before every Bash call cannot
afford what an arm that fires once a turn can.

WHY THE DEFAULTS BELOW ARE STARTING POINTS AND NOT ANSWERS

A cosine score is a distance in BAAI/bge-small-en-v1.5's vector space, measured
against THIS corpus. It cannot transfer to an install with different records,
and rule 115 forbids defending a shipped default from this instance's
telemetry — which is what made the old design unbuildable: every number was a
guess everywhere except here, and there was no mechanism that could ever
improve it.

The mechanism is the fix. Scribe ships a starting point and the means to
correct it, so the values below carry the measurement that motivated them (see
the long comments in `plugin_context.py`, which are kept where they are because
they record how each number was first arrived at) without claiming to be right
for anybody else.

HOW A FLOOR SHOULD ACTUALLY BE MOVED

By reading the records the floor refused — `retrieval_telemetry(
near_miss_samples=N)` returns them by id — and never by the percentile alone.
That is not a style preference; it is the one case where the two disagreed and
was checked. `report_preference` logged 69 consecutive declines with the
refused score 0.0006 under the bar, and every percentile said "lower it".
Reading the refused record showed it was rule 77 "Extract intent from loose
phrasing", a false positive, so lowering the bar would have delivered that rule
on every completion report ever written. The statistic and the correct action
pointed in opposite directions, and only opening the record could tell.
"""
from __future__ import annotations

from dataclasses import dataclass

from scribe.services.settings import bounded_float, get_setting

# A budget nobody should be able to set past. Not a tuning value — a guard on
# the worst case, so a mistyped setting cannot turn a menu into a wall of text.
# Shared by every surface because it bounds the same thing everywhere: how many
# lines a single unsolicited injection may occupy.
MAX_BUDGET = 10


@dataclass(frozen=True)
class Surface:
    """One push arm's tunable pair, plus enough prose to tune it responsibly.

    `asks` / `over` / `fires` are not documentation for this file — they are
    rendered by the tuning tool and the Settings UI. A floor cannot be moved
    sensibly by anyone, model or human, who does not know what the query is, what
    corpus it runs against, or how often it costs something. Those three facts
    are exactly what separates these arms from each other, and they were
    previously recoverable only by reading `plugin_context.py`.
    """

    name: str
    """The telemetry `source` value, and the join key.

    MUST equal the string this arm passes to `record_retrieval`. Everything
    useful about tuning depends on that identity: the tool that moves a floor
    and the table that says what the floor did have to be talking about the same
    arm. A test asserts it rather than a comment asking nicely.
    """

    floor_key: str
    floor_default: float
    budget_key: str
    budget_default: int
    asks: str
    over: str
    fires: str
    measured_model: str = "BAAI/bge-small-en-v1.5"
    measured_shape: int = 1
    """What the SHIPPED defaults above were measured against (#4104).

    A floor is a distance in one embedding model's geometry, over documents cut
    one particular way. Either can change, and when one does every number in
    this table describes something that no longer exists.

    TWO FIELDS, NEVER ONE FUSED STRING (rule 149). A mismatch has to be able to
    say WHICH half moved: a new embedding model and a re-cut document shape
    invalidate the same numbers for different reasons and call for different
    responses. `"<model>@<n>"` could only report that something changed, which
    is the answer nobody can act on. Same reason `calibration_stamp()` returns
    a dict and the event table gives each half its own column.

    Recorded per surface rather than once for the module because they need not
    move together: a surface retuned after a model change carries the new stamp
    while its untouched siblings still carry the old one, and telling those
    apart is the whole job.

    LITERALS, deliberately, rather than an import of the live values — a stamp
    says what was true when the number was chosen, so one that tracked the
    current model would always agree with it and could never report staleness.
    """

    budget_falls_back_to: str = ""
    """A budget key to inherit when this surface has none of its own set.

    Only `write_path` uses it, and only because it USED to share auto-inject's
    `top_k` outright. Giving it a key without this would silently reset the
    budget of every install that had tuned the shared one — a behaviour change
    delivered as a default, which is the shape of regression nobody reports
    because nothing looks broken.
    """


SURFACES: dict[str, Surface] = {
    "auto_inject": Surface(
        name="auto_inject",
        floor_key="kb_autoinject_threshold",
        floor_default=0.55,
        budget_key="kb_autoinject_top_k",
        budget_default=3,
        asks="the operator's message, as they typed it",
        over="notes, snippets, processes and issues",
        fires="once per operator turn",
    ),
    "write_path": Surface(
        name="write_path",
        floor_key="kb_writepath_threshold",
        floor_default=0.68,
        budget_key="kb_writepath_top_k",
        budget_default=3,
        budget_falls_back_to="kb_autoinject_top_k",
        asks="the code being written, rewritten as a concept query",
        over="snippets and recorded issues",
        fires="before every Write and Edit",
    ),
    "write_path_rule": Surface(
        name="write_path_rule",
        floor_key="kb_rulehint_threshold",
        floor_default=0.72,
        budget_key="kb_rulehint_top_k",
        budget_default=5,
        asks="the code being written, against rule triggers",
        over="global rules plus the bound project's own",
        fires="before every Write and Edit",
    ),
    "pre_tool_rule": Surface(
        name="pre_tool_rule",
        floor_key="kb_toolrule_threshold",
        floor_default=0.68,
        budget_key="kb_toolrule_top_k",
        budget_default=5,
        asks="the command about to run, against rule triggers",
        over="global rules plus the bound project's own",
        fires="before every Bash call — the busiest arm there is",
    ),
    "prompt_rule": Surface(
        name="prompt_rule",
        floor_key="kb_promptrule_threshold",
        floor_default=0.72,
        budget_key="kb_promptrule_top_k",
        budget_default=3,
        asks="the operator's message, against rule triggers",
        over="global rules plus the bound project's own",
        fires="once per operator turn",
    ),
    "report_preference": Surface(
        name="report_preference",
        floor_key="kb_reportpref_threshold",
        floor_default=0.72,
        budget_key="kb_reportpref_top_k",
        budget_default=3,
        # THE ONE FIXED QUERY, and the reason this arm behaves unlike the rest.
        # The others score something that varies per call; this one scores a
        # constant string, so its top score for a given corpus is also a
        # constant. A floor a hair above that constant is not a quiet arm, it is
        # a dead one, and no amount of traffic will ever reveal it — which is
        # precisely how this arm spent 69 calls declining the same record.
        asks="a fixed question about how to lay out a completion report",
        over="preferences",
        fires="when a task finishes",
    ),
}

# Reserved slots are deliberately absent. `preference_slot`, `reuse_slot` and
# `lesson_slot` borrow their parent arm's floor and are hard-limited to one hit
# each, because their entire purpose is to guarantee a single line to a kind of
# record that keeps losing a general score contest (#2246, #3894) — or, for
# `lesson_slot`, to a kind whose loss is total rather than merely a lost
# convenience, since a lesson has no act arm to fall back on and nobody browses
# lessons looking for one (milestone 385). A budget of "1" is the feature;
# exposing it as tunable would invite setting it to 0 and silently removing the
# guarantee.
#
# They are still logged under their own `source`, which is what keeps them
# judgeable without being tunable: `best_available_id` names the record each
# bar refused, so a slot that never places, or one that places weak hits, shows
# up as evidence rather than as an argument.


def surface_names() -> list[str]:
    """Every tunable surface, in a stable order for menus and listings."""
    return list(SURFACES)


def get_surface(name: str) -> Surface:
    """Look one up, refusing an unknown name loudly.

    A typo'd surface must not be writable. Settings keys are free-form strings
    in a generic table, so a tuning call naming `pretool_rule` would otherwise
    write a key nothing ever reads — a change that appears to succeed, reports a
    new value, and alters nothing.
    """
    try:
        return SURFACES[name]
    except KeyError:
        raise ValueError(
            f"unknown retrieval surface {name!r}. Tunable surfaces are: "
            + ", ".join(surface_names())
        ) from None


def dial_for_key(key: str) -> tuple[str, str] | None:
    """Which `(surface, dial)` a settings key belongs to, or None.

    The registry read backwards, and it exists for one caller: the generic
    `/api/settings` endpoint, which accepts any key at all. Without this, a
    floor written through that endpoint moves with no event recorded, and the
    tuning history says nothing happened — a trail with holes in it, which is
    worse than no trail because it reads as complete.

    Derived rather than listed so a seventh surface is covered the moment it is
    added here, which is the only way this stays true.
    """
    for surface in SURFACES.values():
        if key == surface.floor_key:
            return (surface.name, "floor")
        if key == surface.budget_key:
            return (surface.name, "budget")
    return None


async def floor_for(user_id: int, name: str) -> float:
    """This install's current floor for a surface, clamped to [0, 1]."""
    s = get_surface(name)
    raw = await get_setting(user_id, s.floor_key, str(s.floor_default))
    return bounded_float(raw, s.floor_default)


async def budget_for(user_id: int, name: str) -> int:
    """This install's current budget for a surface, clamped to [1, MAX_BUDGET].

    The lower clamp is 1, never 0: a surface turned off is turned off by its
    `enabled` switch, which says so. A budget of zero would be an arm that runs
    a search, logs a retrieval, and renders nothing — indistinguishable in the
    telemetry from a bar nothing cleared, which is the exact confusion this
    milestone exists to remove.
    """
    s = get_surface(name)
    raw = await get_setting(user_id, s.budget_key, "")
    if not raw and s.budget_falls_back_to:
        raw = await get_setting(user_id, s.budget_falls_back_to, "")
    try:
        value = int(float(raw)) if raw else s.budget_default
    except (TypeError, ValueError):
        value = s.budget_default
    return min(MAX_BUDGET, max(1, value))