bvandeusenandClaude Opus 5.5
7c1d98dd2a
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m2s
CI & Build / Python tests (push) Failing after 1m26s
CI & Build / Build & push image (push) Skipped
A rule arm logs its query, floor, budget and project, which is everything semantic_search_rules takes, so a logged call is re-run to a depth past its budget and judged like an auto_inject menu. within_budget is the arm's own cut: inside the budget and, for the banded command and write arms, inside the band of the top hit. "Opened after" reads rule pulls, and rules created after the call are dropped as notes are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
473 lines
20 KiB
Python
473 lines
20 KiB
Python
"""Judging what a retrieval arm offered — the review pass.
|
|
|
|
The usage tables record what the reader DID with a line: opened it or not.
|
|
Because every menu line carries its matched passage, that action does not
|
|
stand in for relevance — a line can be unrelated, related and enough
|
|
as shown, related and already in context, or related and ignored, and all four
|
|
are "not opened". Only someone reading the query beside the line can tell
|
|
them apart. This module is how that reading gets done and kept.
|
|
|
|
THE SHAPE. `retrieval_logs` already stores each call's query (enriched, as the
|
|
arm searched it) and its ranked candidates. A review takes a sample of logged
|
|
calls the reviewer has not judged, RE-RUNS each query with the arm's own
|
|
parameters to a depth past its budget, and hands back every candidate with the
|
|
passage that matched. The reviewer records a verdict per line, with a reason.
|
|
|
|
WHY RE-RUN rather than replay the logged ids. The log holds ids and scores,
|
|
not passages, and the reviewer needs the passage — it is what the reader saw.
|
|
And the depth past the budget is the point: the candidates just under the cut
|
|
are the ones a budget change would add, and the only way to judge them is to
|
|
look at them. The re-run is against today's corpus; `missing` names any
|
|
logged id the re-run no longer finds, so drift is visible rather than silent.
|
|
|
|
WHAT TODAY'S CORPUS GETS WRONG. A record written AFTER the call could not have
|
|
been offered to it, yet it outranks everything: the session that made the call
|
|
goes on to write the decision, the fix, the work log — often quoting the very
|
|
message. Left in, it reads as the arm's best hit and inflates `on_point`
|
|
exactly where the budget question is decided. So the re-run drops records
|
|
created after the call (`postdated` names them), and marks a line
|
|
`changed_since_call` when the record or one of its work logs was written after
|
|
it — its passage may be text the reader never saw.
|
|
|
|
WHY IT WRITES NO TELEMETRY. A review is not a retrieval surface — nothing is
|
|
put in front of a working session — so it records no `retrieval_logs` row and
|
|
no surfacing. A row here would be a call to the arm that the arm never made.
|
|
|
|
ATTENDED, NEVER BULK. Every verdict needs a reason. A verdict without one is a
|
|
threshold with extra steps, which is the thing this exists to replace.
|
|
"""
|
|
import logging
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
from sqlalchemy import and_, exists, func, select
|
|
|
|
from grimoire.models import async_session
|
|
from grimoire.models.base import iso
|
|
from grimoire.models.note_usage import PULLED, NoteUsageEvent
|
|
from grimoire.models.retrieval_judgment import VERDICTS, RetrievalJudgment
|
|
from grimoire.models.retrieval_log import RetrievalLog
|
|
from grimoire.models.rule_usage import PULLED as RULE_PULLED
|
|
from grimoire.models.rule_usage import RuleUsageEvent
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# The arms a review can re-run FAITHFULLY. A re-run with different parameters
|
|
# from the arm's would judge a menu nobody was shown, so a surface joins this
|
|
# only with its own search spelled out in `_rerun`. The rule arms named here
|
|
# are the ones that annotate: each logs its query, floor, budget and project,
|
|
# which is everything its search takes.
|
|
RULE_REVIEWABLE = ("prompt_rule", "pre_tool_rule", "write_path_rule")
|
|
REVIEWABLE = ("auto_inject", *RULE_REVIEWABLE)
|
|
|
|
DEPTH_DEFAULT = 6
|
|
DEPTH_MAX = 10
|
|
SAMPLE_MAX = 20
|
|
# Extra candidates fetched so that dropping post-dated records still leaves
|
|
# `depth` lines. Records written after a call cluster at the top of its
|
|
# re-run, so the slack has to cover a session's worth of them.
|
|
POSTDATED_SLACK = 15
|
|
# How long after a call a pull still counts as "opened after". The server has
|
|
# no session identity (NoteUsageEvent), so this is a window, not a join.
|
|
OPENED_WINDOW = timedelta(hours=1)
|
|
|
|
HOW_TO_JUDGE = (
|
|
"Read each line as the reader did: the query, then the line's name, kind "
|
|
"and passage. Judge from those — do NOT open the record, which counts as a "
|
|
"pull and would credit the arm with an open it never earned. Verdicts: "
|
|
"`on_point` — someone in the query's situation should read this; "
|
|
"`adjacent` — same area, but it would not change what they do; "
|
|
"`unrelated` — a near neighbour in words only. Every verdict needs a "
|
|
"`reason` naming what in the query and the passage decided it. Judge the "
|
|
"lines past the budget too (`within_budget: false`): they are the "
|
|
"candidates a larger budget would add. A line marked `changed_since_call` "
|
|
"was written to after the call: if its passage is plainly the later text "
|
|
"(dated after the call, or about what the call led to), leave it unjudged "
|
|
"— the reader never saw it."
|
|
)
|
|
|
|
|
|
def _check_source(source: str) -> str:
|
|
if source not in REVIEWABLE:
|
|
raise ValueError(f"source must be one of {list(REVIEWABLE)}, got {source!r}")
|
|
return source
|
|
|
|
|
|
def _after(stamp, call) -> bool:
|
|
return stamp is not None and call is not None and stamp > call
|
|
|
|
|
|
async def _rerun(
|
|
user_id: int, row: RetrievalLog, depth: int, report: dict | None = None,
|
|
) -> list[dict]:
|
|
"""The logged call's query, searched again the way its arm searches."""
|
|
if row.source in RULE_REVIEWABLE:
|
|
return await _rerun_rules(user_id, row, depth, report)
|
|
return await _rerun_notes(user_id, row, depth, report)
|
|
|
|
|
|
async def _rerun_rules(
|
|
user_id: int, row: RetrievalLog, depth: int, report: dict | None = None,
|
|
) -> list[dict]:
|
|
"""A rule arm's call, searched again: its query against rule triggers,
|
|
with the floor and project it logged.
|
|
|
|
`within_budget` is the arm's own cut: a line inside the budget, and, for
|
|
an arm that bands, inside the band of the top hit. The preference slot is
|
|
left out — it logs its own source. Rules created after the call are
|
|
dropped and named in `report["postdated"]`, as for notes.
|
|
"""
|
|
from grimoire.services import retrieval_pipeline as rp
|
|
from grimoire.services.embeddings import semantic_search_rules
|
|
|
|
arm = next(a for a in rp.RULE_ARMS if a.source == row.source)
|
|
hits = await semantic_search_rules(
|
|
user_id, row.query or "",
|
|
limit=depth + POSTDATED_SLACK,
|
|
threshold=row.threshold if row.threshold is not None else 0.0,
|
|
project_id=row.project_id or None,
|
|
)
|
|
postdated = [int(r.id) for _s, r in hits if _after(r.created_at, row.created_at)]
|
|
if report is not None:
|
|
report["postdated"] = postdated
|
|
hits = [h for h in hits if int(h[1].id) not in postdated][:depth]
|
|
budget = int(row.limit_n or 0)
|
|
cut = hits[:budget]
|
|
kept = {int(r.id) for _s, r in (rp._rule_band(cut) if arm.band else cut)}
|
|
return [{
|
|
"rank": rank,
|
|
"record_id": int(rule.id),
|
|
"score": round(float(score), 4),
|
|
"within_budget": int(rule.id) in kept,
|
|
"changed_since_call": _after(rule.updated_at, row.created_at),
|
|
"kind": rule.kind or "rule",
|
|
"name": rule.title,
|
|
# The trigger is what the arm matched against and what its line
|
|
# quotes, so it is the passage the reader saw.
|
|
"passage": rule.when_to_apply or rule.statement or None,
|
|
} for rank, (score, rule) in enumerate(hits, start=1)]
|
|
|
|
|
|
async def _rerun_notes(
|
|
user_id: int, row: RetrievalLog, depth: int, report: dict | None = None,
|
|
) -> list[dict]:
|
|
"""A notes arm's call, searched again.
|
|
|
|
`auto_inject`'s parameters, read from its spec (`retrieval_pipeline.
|
|
AUTO_INJECT`) rather than restated: which kinds, whose records, and the
|
|
arm's floor as logged. The reserved slots (reuse, lesson) are left out —
|
|
each logs its own source, and is judged as that source when it becomes
|
|
reviewable.
|
|
|
|
Records created after the call are dropped before ranking — the call could
|
|
not have been offered them — and named in `report["postdated"]`.
|
|
"""
|
|
# Imported here: plugin_context imports retrieval_telemetry, which reads
|
|
# this module's judged block.
|
|
from grimoire.services import retrieval_pipeline as rp
|
|
from grimoire.services.embeddings import document_title, semantic_search_notes
|
|
from grimoire.services.retrieval_pipeline import _menu_name, _menu_passage, _record_kind
|
|
|
|
rep: dict = {}
|
|
hits = await semantic_search_notes(
|
|
user_id, row.query or "",
|
|
limit=depth + POSTDATED_SLACK,
|
|
threshold=row.threshold if row.threshold is not None else 0.0,
|
|
project_id=row.project_id or None,
|
|
**rp.note_search_filters(rp.AUTO_INJECT),
|
|
report=rep,
|
|
)
|
|
postdated = [int(n.id) for _s, n in hits if _after(n.created_at, row.created_at)]
|
|
if report is not None:
|
|
report["postdated"] = postdated
|
|
hits = [h for h in hits if int(h[1].id) not in postdated][:depth]
|
|
chunks = rep.get("best_chunk") or {}
|
|
budget = int(row.limit_n or 0)
|
|
out = []
|
|
for rank, (score, note) in enumerate(hits, start=1):
|
|
name = _menu_name(note.title, note.note_type, note.data, note.body)
|
|
out.append({
|
|
"rank": rank,
|
|
"record_id": int(note.id),
|
|
"score": round(float(score), 4),
|
|
"within_budget": rank <= budget,
|
|
# Work logs are folded in by menus_to_review, which has a session.
|
|
"changed_since_call": _after(note.updated_at, row.created_at),
|
|
"kind": _record_kind(note),
|
|
"name": name,
|
|
"passage": _menu_passage(
|
|
document_title(note.title, note.note_type, note.data, note.body),
|
|
(chunks.get(int(note.id)) or {}).get("text"), name,
|
|
) or None,
|
|
})
|
|
return out
|
|
|
|
|
|
async def _opened_after(user_id: int, row: RetrievalLog, ids: list[int]) -> set[int]:
|
|
"""Which of `ids` the user's AGENT pulled within OPENED_WINDOW of the call.
|
|
|
|
Agent pulls only — the `mcp_` prefix, per NoteUsageEvent's convention —
|
|
because the question is whether the line moved the session, not whether
|
|
the operator later browsed to the record. A rule arm's lines are read
|
|
from the rule table: a note and a rule can share an id.
|
|
"""
|
|
if not ids or row.created_at is None:
|
|
return set()
|
|
table, id_col, pulled = (
|
|
(RuleUsageEvent, RuleUsageEvent.rule_id, RULE_PULLED)
|
|
if row.source in RULE_REVIEWABLE
|
|
else (NoteUsageEvent, NoteUsageEvent.note_id, PULLED)
|
|
)
|
|
async with async_session() as session:
|
|
found = await session.execute(
|
|
select(id_col).distinct().where(
|
|
table.user_id == user_id,
|
|
table.event == pulled,
|
|
table.source.startswith("mcp_", autoescape=True),
|
|
id_col.in_(ids),
|
|
table.created_at >= row.created_at,
|
|
table.created_at <= row.created_at + OPENED_WINDOW,
|
|
)
|
|
)
|
|
return {int(i) for (i,) in found.all()}
|
|
|
|
|
|
async def _logged_after(row: RetrievalLog, ids: list[int]) -> set[int]:
|
|
"""Which of `ids` gained a work log after the call. A log lands in its
|
|
task's embedded document without always touching the task's `updated_at`
|
|
(a closed task takes no claim stamp), so it is read here."""
|
|
from grimoire.models.task_log import TaskLog
|
|
|
|
if not ids or row.created_at is None or row.source in RULE_REVIEWABLE:
|
|
return set()
|
|
async with async_session() as session:
|
|
found = await session.execute(
|
|
select(TaskLog.task_id).distinct().where(
|
|
TaskLog.task_id.in_(ids), TaskLog.created_at > row.created_at,
|
|
)
|
|
)
|
|
return {int(i) for (i,) in found.all()}
|
|
|
|
|
|
def _eligible(user_id: int, source: str, since):
|
|
"""Logged calls of `source` that offered something fresh and that this
|
|
reviewer has not judged a line of."""
|
|
judged = exists().where(and_(
|
|
RetrievalJudgment.retrieval_log_id == RetrievalLog.id,
|
|
RetrievalJudgment.user_id == user_id,
|
|
))
|
|
return (
|
|
RetrievalLog.user_id == user_id,
|
|
RetrievalLog.source == source,
|
|
RetrievalLog.created_at >= since,
|
|
RetrievalLog.result_count > 0,
|
|
RetrievalLog.query.is_not(None),
|
|
~judged,
|
|
)
|
|
|
|
|
|
async def menus_to_review(
|
|
user_id: int, *, source: str = "auto_inject", n: int = 5, days: int = 14,
|
|
depth: int = DEPTH_DEFAULT,
|
|
) -> dict:
|
|
"""A random sample of unjudged logged calls, each re-run for judging.
|
|
|
|
Random, not newest: a review of the latest calls judges one afternoon's
|
|
work, and the readout would then describe that afternoon.
|
|
"""
|
|
_check_source(source)
|
|
n = max(1, min(int(n), SAMPLE_MAX))
|
|
depth = max(1, min(int(depth), DEPTH_MAX))
|
|
since = datetime.now(timezone.utc) - timedelta(days=max(1, int(days)))
|
|
where = _eligible(user_id, source, since)
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(RetrievalLog).where(*where).order_by(func.random()).limit(n)
|
|
)).scalars().all()
|
|
remaining = (await session.execute(
|
|
select(func.count()).select_from(RetrievalLog).where(*where)
|
|
)).scalar_one()
|
|
|
|
menus = []
|
|
for row in rows:
|
|
rep: dict = {}
|
|
lines = await _rerun(user_id, row, depth, report=rep)
|
|
ids = [ln["record_id"] for ln in lines]
|
|
opened = await _opened_after(user_id, row, ids)
|
|
relogged = await _logged_after(row, ids)
|
|
logged = [int(it["id"]) for it in (row.result_ids or [])]
|
|
for ln in lines:
|
|
ln["opened_after"] = ln["record_id"] in opened
|
|
ln["changed_since_call"] = ln["changed_since_call"] or ln["record_id"] in relogged
|
|
# Shown fresh then. Inside the budget and not logged means a
|
|
# repeat the arm withheld, or a ranking the corpus has since moved.
|
|
ln["logged"] = ln["record_id"] in logged
|
|
rerun_ids = set(ids)
|
|
menus.append({
|
|
"log_id": int(row.id),
|
|
"created_at": iso(row.created_at),
|
|
"query": row.query,
|
|
"budget": row.limit_n,
|
|
"threshold": row.threshold,
|
|
"logged_ids": logged,
|
|
# Logged then, not found now: deleted, edited out of reach, or
|
|
# outranked since. Judge what is here.
|
|
"missing": [i for i in logged if i not in rerun_ids],
|
|
# Written after the call, so never offered to it; left out.
|
|
"postdated": rep.get("postdated", []),
|
|
"lines": lines,
|
|
})
|
|
return {
|
|
"source": source,
|
|
"menus": menus,
|
|
"remaining": int(remaining),
|
|
"verdicts": list(VERDICTS),
|
|
"how_to_judge": HOW_TO_JUDGE,
|
|
}
|
|
|
|
|
|
async def judge_menu(
|
|
user_id: int, *, log_id: int, verdicts: list[dict],
|
|
) -> dict:
|
|
"""Record a verdict, with its reason, on lines of one logged call.
|
|
|
|
Rank, score and budget side are re-derived from a fresh re-run rather than
|
|
taken from the caller, so a verdict cannot be filed against a position the
|
|
record does not hold. Judging a line again replaces the earlier verdict.
|
|
"""
|
|
if not verdicts:
|
|
raise ValueError("pass at least one verdict")
|
|
parsed: list[tuple[int, str, str]] = []
|
|
for v in verdicts:
|
|
rid = int(v.get("record_id") or 0)
|
|
verdict = str(v.get("verdict") or "").strip().lower()
|
|
reason = str(v.get("reason") or "").strip()
|
|
if verdict not in VERDICTS:
|
|
raise ValueError(f"verdict must be one of {list(VERDICTS)}, got {verdict!r}")
|
|
if not reason:
|
|
raise ValueError(
|
|
f"record {rid}: a verdict needs its reason — what in the query "
|
|
f"and the passage decided it. Without one it cannot be re-read."
|
|
)
|
|
parsed.append((rid, verdict, reason))
|
|
|
|
async with async_session() as session:
|
|
row = await session.get(RetrievalLog, int(log_id))
|
|
# Another user's call reads exactly like a missing one: its query is theirs.
|
|
if row is None or row.user_id != user_id:
|
|
raise ValueError(f"logged call {log_id} not found")
|
|
_check_source(row.source)
|
|
|
|
lines = {ln["record_id"]: ln for ln in await _rerun(user_id, row, DEPTH_MAX)}
|
|
unknown = [rid for rid, _v, _r in parsed if rid not in lines]
|
|
if unknown:
|
|
raise ValueError(
|
|
f"records {unknown} are not in this call's re-run to depth "
|
|
f"{DEPTH_MAX} — judge the lines menus_to_review returned"
|
|
)
|
|
opened = await _opened_after(user_id, row, [rid for rid, _v, _r in parsed])
|
|
|
|
async with async_session() as session:
|
|
existing = {
|
|
int(j.record_id): j
|
|
for j in (await session.execute(
|
|
select(RetrievalJudgment).where(
|
|
RetrievalJudgment.retrieval_log_id == row.id,
|
|
RetrievalJudgment.user_id == user_id,
|
|
RetrievalJudgment.record_id.in_([rid for rid, _v, _r in parsed]),
|
|
)
|
|
)).scalars().all()
|
|
}
|
|
for rid, verdict, reason in parsed:
|
|
ln = lines[rid]
|
|
j = existing.get(rid) or RetrievalJudgment(
|
|
user_id=user_id, retrieval_log_id=int(row.id), record_id=rid,
|
|
)
|
|
j.source = row.source
|
|
j.query = row.query
|
|
j.rank = ln["rank"]
|
|
j.score = ln["score"]
|
|
j.within_budget = ln["within_budget"]
|
|
j.verdict = verdict
|
|
j.reason = reason
|
|
j.opened_after = rid in opened
|
|
session.add(j)
|
|
await session.commit()
|
|
return {
|
|
"log_id": int(row.id),
|
|
"recorded": len(parsed),
|
|
"replaced": len(existing),
|
|
}
|
|
|
|
|
|
async def judged_block(user_id: int | None, since) -> dict:
|
|
"""`retrieval_telemetry`'s judged block: verdicts by rank, per source.
|
|
|
|
Counts, not rates, beside each other per rank — the sample is small by
|
|
nature and a percentage of seven reads as more than it is. `on_point_unopened`
|
|
is the number the open rate could never show: lines that related and were
|
|
left closed, which is the passage doing its job or the reader missing it.
|
|
|
|
Raises on a failed read; the caller guards it.
|
|
"""
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(
|
|
RetrievalJudgment.source,
|
|
RetrievalJudgment.rank,
|
|
RetrievalJudgment.within_budget,
|
|
RetrievalJudgment.verdict,
|
|
RetrievalJudgment.opened_after,
|
|
func.count().label("n"),
|
|
)
|
|
.where(
|
|
RetrievalJudgment.user_id == user_id,
|
|
RetrievalJudgment.created_at >= since,
|
|
)
|
|
.group_by(
|
|
RetrievalJudgment.source, RetrievalJudgment.rank,
|
|
RetrievalJudgment.within_budget, RetrievalJudgment.verdict,
|
|
RetrievalJudgment.opened_after,
|
|
)
|
|
)).all()
|
|
calls = dict((await session.execute(
|
|
select(
|
|
RetrievalJudgment.source,
|
|
func.count(func.distinct(RetrievalJudgment.retrieval_log_id)),
|
|
)
|
|
.where(
|
|
RetrievalJudgment.user_id == user_id,
|
|
RetrievalJudgment.created_at >= since,
|
|
)
|
|
.group_by(RetrievalJudgment.source)
|
|
)).all())
|
|
|
|
def empty() -> dict:
|
|
return {v: 0 for v in VERDICTS} | {"opened": 0}
|
|
|
|
out: dict = {}
|
|
for source, rank, within, verdict, opened, n in rows:
|
|
block = out.setdefault(source, {
|
|
"judged_calls": int(calls.get(source, 0)),
|
|
"judged_lines": 0,
|
|
"within_budget": empty(),
|
|
"beyond_budget": empty(),
|
|
"by_rank": {},
|
|
"on_point_unopened": 0,
|
|
})
|
|
n = int(n)
|
|
block["judged_lines"] += n
|
|
side = block["within_budget" if within else "beyond_budget"]
|
|
at = block["by_rank"].setdefault(int(rank), empty())
|
|
for bucket in (side, at):
|
|
if verdict in VERDICTS:
|
|
bucket[verdict] += n
|
|
if opened:
|
|
bucket["opened"] += n
|
|
if verdict == VERDICTS[0] and not opened:
|
|
block["on_point_unopened"] += n
|
|
for block in out.values():
|
|
block["by_rank"] = [
|
|
{"rank": r, **c} for r, c in sorted(block["by_rank"].items())
|
|
]
|
|
return out
|