bvandeusenandClaude Opus 5.5
7c1d98dd2a
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m2s
CI & Build / Python tests (push) Failing after 1m26s
CI & Build / Build & push image (push) Skipped
A rule arm logs its query, floor, budget and project, which is everything semantic_search_rules takes, so a logged call is re-run to a depth past its budget and judged like an auto_inject menu. within_budget is the arm's own cut: inside the budget and, for the banded command and write arms, inside the band of the top hit. "Opened after" reads rule pulls, and rules created after the call are dropped as notes are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
86 lines
4.0 KiB
Python
86 lines
4.0 KiB
Python
"""Judging what a retrieval arm offered.
|
||
|
||
`retrieval_telemetry` says what the reader did with each line — opened it or
|
||
not. These two say whether the line RELATED, which is a reading only someone
|
||
looking at the query beside the line can do.
|
||
"""
|
||
from grimoire.mcp._context import current_user_id
|
||
from grimoire.services import retrieval_review as review_svc
|
||
|
||
|
||
async def menus_to_review(
|
||
source: str = "auto_inject", n: int = 5, days: int = 14, depth: int = 6,
|
||
) -> dict:
|
||
"""A sample of logged injection menus, re-run so you can judge each line.
|
||
|
||
Reach for this before moving a budget or a floor, and whenever an open rate
|
||
is about to be read as a verdict on relevance. It is not one: every menu
|
||
line carries the passage that matched, so a record left unopened may have
|
||
been unrelated, enough as shown, or already in context, and the usage
|
||
counters cannot tell those apart. A judged sample can.
|
||
|
||
Each menu is one logged call: the query the arm searched (enriched, as it
|
||
was asked), its `budget`, and `lines` — the query re-run with the arm's own
|
||
parameters to `depth`, each line with `rank`, `score`, `kind`, `name`, the
|
||
`passage` that matched, `within_budget` (was it inside the cut) and
|
||
`opened_after` (did your agent open it within an hour). Lines past the
|
||
budget are there on purpose: they are what a larger budget would add.
|
||
|
||
`missing` names ids the call logged that the re-run no longer finds — the
|
||
corpus has moved since. Judge what is in front of you.
|
||
|
||
The re-run is against today's corpus, so it guards what has been written
|
||
since. Records created after the call are left out and named in
|
||
`postdated`: the call could not have offered them, and the session that
|
||
made it often wrote them, quoting it. A line marked `changed_since_call`
|
||
was written to afterwards; if its passage is plainly the later text, leave
|
||
it unjudged. `logged` says the line was shown fresh at the time.
|
||
|
||
JUDGE FROM THE LINE; DO NOT OPEN THE RECORD. Opening counts as a pull and
|
||
would credit the arm with an open it never earned. Then record verdicts
|
||
with `judge_menu`. `how_to_judge` in the response repeats the vocabulary.
|
||
|
||
Args:
|
||
source: the arm to review. Only arms whose search can be re-run
|
||
exactly are offered: `auto_inject`, and the rule arms
|
||
`prompt_rule`, `pre_tool_rule` and `write_path_rule`. A rule
|
||
line's passage is the trigger the arm matched, and for the
|
||
command and write arms `within_budget` also needs the line to
|
||
sit inside the band of the top hit, as it did when shown.
|
||
n: how many calls to sample, 1–20. Random among the calls you have not
|
||
judged, so a review does not describe one afternoon.
|
||
days: how far back to sample from.
|
||
depth: how many candidates to re-run per call, 1–10. Past the budget,
|
||
so the cut itself can be judged.
|
||
"""
|
||
return await review_svc.menus_to_review(
|
||
current_user_id(), source=source, n=n, days=days, depth=depth,
|
||
)
|
||
|
||
|
||
async def judge_menu(log_id: int, verdicts: list[dict]) -> dict:
|
||
"""Record your verdict on lines of one menu from `menus_to_review`.
|
||
|
||
`verdicts` is a list of `{record_id, verdict, reason}`:
|
||
|
||
"on_point" — someone in the query's situation should read this.
|
||
"adjacent" — the same area, but it would not change what they do.
|
||
"unrelated" — close in words only.
|
||
|
||
`reason` is REQUIRED for every verdict: what in the query and the passage
|
||
decided it. A verdict nobody can re-read is a threshold, and this exists to
|
||
replace one. Rank, score and which side of the budget the line sat are
|
||
taken from a fresh re-run, not from you. Judging a line again replaces your
|
||
earlier verdict on it.
|
||
|
||
The verdicts are read back as `retrieval_telemetry`'s `judged` block.
|
||
"""
|
||
return await review_svc.judge_menu(
|
||
current_user_id(), log_id=log_id, verdicts=verdicts,
|
||
)
|
||
|
||
|
||
def register(mcp) -> None:
|
||
mcp.tool(name="menus_to_review")(menus_to_review)
|
||
mcp.tool(name="judge_menu")(judge_menu)
|