rename(547): the server is grimoire — package, image, DB defaults, backups

Step 2 of the Scribe -> Grimoire rename.

- src/scribe -> src/grimoire, and every import, patch target, module path,
  hypercorn entry point and `python -m grimoire.restore` with it; pyproject,
  uv.lock and the frontend package are named grimoire.
- Image git.fabledsword.com/bvandeusen/fabledgrimoire; CI's Postgres service
  runs as grimoire / grimoire_test.
- Compose files default the DB user, database and (prod) volume to grimoire,
  each overridable from .env, so an install created as Scribe keeps its data
  by setting POSTGRES_USER, POSTGRES_DB and PGDATA_VOLUME.
- Backups are written as grimoire-<ts>.tar with format grimoire-backup-tar.
  Archives named scribe-* with the scribe-backup-tar manifest still restore
  and are still counted at the destination (folder and S3), with tests.
- Exports, the MCP auth realm, the ASGI scope key and the persistent logger
  carry the new name.

Kept on purpose: the secret-sealing key label stays "scribe.secrets.v1",
because every stored secret was derived under it.

Plugin identity (plugin name, hooks, env vars, .scribe marker, skills) is
step 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/grimoire/services/retrieval_review.py

@@ -0,0 +1,410 @@
+"""Judging what a retrieval arm offered — the review pass.
+
+The usage tables record what the reader DID with a line: opened it or not.
+Because every menu line carries its matched passage, that action does not
+stand in for relevance — a line can be unrelated, related and enough
+as shown, related and already in context, or related and ignored, and all four
+are "not opened". Only someone reading the query beside the line can tell
+them apart. This module is how that reading gets done and kept.
+
+THE SHAPE. `retrieval_logs` already stores each call's query (enriched, as the
+arm searched it) and its ranked candidates. A review takes a sample of logged
+calls the reviewer has not judged, RE-RUNS each query with the arm's own
+parameters to a depth past its budget, and hands back every candidate with the
+passage that matched. The reviewer records a verdict per line, with a reason.
+
+WHY RE-RUN rather than replay the logged ids. The log holds ids and scores,
+not passages, and the reviewer needs the passage — it is what the reader saw.
+And the depth past the budget is the point: the candidates just under the cut
+are the ones a budget change would add, and the only way to judge them is to
+look at them. The re-run is against today's corpus; `missing` names any
+logged id the re-run no longer finds, so drift is visible rather than silent.
+
+WHAT TODAY'S CORPUS GETS WRONG. A record written AFTER the call could not have
+been offered to it, yet it outranks everything: the session that made the call
+goes on to write the decision, the fix, the work log — often quoting the very
+message. Left in, it reads as the arm's best hit and inflates `on_point`
+exactly where the budget question is decided. So the re-run drops records
+created after the call (`postdated` names them), and marks a line
+`changed_since_call` when the record or one of its work logs was written after
+it — its passage may be text the reader never saw.
+
+WHY IT WRITES NO TELEMETRY. A review is not a retrieval surface — nothing is
+put in front of a working session — so it records no `retrieval_logs` row and
+no surfacing. A row here would be a call to the arm that the arm never made.
+
+ATTENDED, NEVER BULK. Every verdict needs a reason. A verdict without one is a
+threshold with extra steps, which is the thing this exists to replace.
+"""
+import logging
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import and_, exists, func, select
+
+from grimoire.models import async_session
+from grimoire.models.base import iso
+from grimoire.models.note_usage import PULLED, NoteUsageEvent
+from grimoire.models.retrieval_judgment import VERDICTS, RetrievalJudgment
+from grimoire.models.retrieval_log import RetrievalLog
+
+logger = logging.getLogger(__name__)
+
+# The arms a review can re-run FAITHFULLY. A re-run with different parameters
+# from the arm's would judge a menu nobody was shown, so a surface joins this
+# only with its own search spelled out in `_rerun`.
+REVIEWABLE = ("auto_inject",)
+
+DEPTH_DEFAULT = 6
+DEPTH_MAX = 10
+SAMPLE_MAX = 20
+# Extra candidates fetched so that dropping post-dated records still leaves
+# `depth` lines. Records written after a call cluster at the top of its
+# re-run, so the slack has to cover a session's worth of them.
+POSTDATED_SLACK = 15
+# How long after a call a pull still counts as "opened after". The server has
+# no session identity (NoteUsageEvent), so this is a window, not a join.
+OPENED_WINDOW = timedelta(hours=1)
+
+HOW_TO_JUDGE = (
+    "Read each line as the reader did: the query, then the line's name, kind "
+    "and passage. Judge from those — do NOT open the record, which counts as a "
+    "pull and would credit the arm with an open it never earned. Verdicts: "
+    "`on_point` — someone in the query's situation should read this; "
+    "`adjacent` — same area, but it would not change what they do; "
+    "`unrelated` — a near neighbour in words only. Every verdict needs a "
+    "`reason` naming what in the query and the passage decided it. Judge the "
+    "lines past the budget too (`within_budget: false`): they are the "
+    "candidates a larger budget would add. A line marked `changed_since_call` "
+    "was written to after the call: if its passage is plainly the later text "
+    "(dated after the call, or about what the call led to), leave it unjudged "
+    "— the reader never saw it."
+)
+
+
+def _check_source(source: str) -> str:
+    if source not in REVIEWABLE:
+        raise ValueError(f"source must be one of {list(REVIEWABLE)}, got {source!r}")
+    return source
+
+
+def _after(stamp, call) -> bool:
+    return stamp is not None and call is not None and stamp > call
+
+
+async def _rerun(
+    user_id: int, row: RetrievalLog, depth: int, report: dict | None = None,
+) -> list[dict]:
+    """The logged call's query, searched again the way its arm searches.
+
+    `auto_inject`'s parameters, read from its spec (`retrieval_pipeline.
+    AUTO_INJECT`) rather than restated: which kinds, whose records, and the
+    arm's floor as logged. The reserved slots (reuse, lesson) are left out —
+    each logs its own source, and is judged as that source when it becomes
+    reviewable.
+
+    Records created after the call are dropped before ranking — the call could
+    not have been offered them — and named in `report["postdated"]`.
+    """
+    # Imported here: plugin_context imports retrieval_telemetry, which reads
+    # this module's judged block.
+    from grimoire.services import retrieval_pipeline as rp
+    from grimoire.services.embeddings import document_title, semantic_search_notes
+    from grimoire.services.retrieval_pipeline import _menu_name, _menu_passage, _record_kind
+
+    rep: dict = {}
+    hits = await semantic_search_notes(
+        user_id, row.query or "",
+        limit=depth + POSTDATED_SLACK,
+        threshold=row.threshold if row.threshold is not None else 0.0,
+        project_id=row.project_id or None,
+        **rp.note_search_filters(rp.AUTO_INJECT),
+        report=rep,
+    )
+    postdated = [int(n.id) for _s, n in hits if _after(n.created_at, row.created_at)]
+    if report is not None:
+        report["postdated"] = postdated
+    hits = [h for h in hits if int(h[1].id) not in postdated][:depth]
+    chunks = rep.get("best_chunk") or {}
+    budget = int(row.limit_n or 0)
+    out = []
+    for rank, (score, note) in enumerate(hits, start=1):
+        name = _menu_name(note.title, note.note_type, note.data, note.body)
+        out.append({
+            "rank": rank,
+            "record_id": int(note.id),
+            "score": round(float(score), 4),
+            "within_budget": rank <= budget,
+            # Work logs are folded in by menus_to_review, which has a session.
+            "changed_since_call": _after(note.updated_at, row.created_at),
+            "kind": _record_kind(note),
+            "name": name,
+            "passage": _menu_passage(
+                document_title(note.title, note.note_type, note.data, note.body),
+                (chunks.get(int(note.id)) or {}).get("text"), name,
+            ) or None,
+        })
+    return out
+
+
+async def _opened_after(user_id: int, row: RetrievalLog, ids: list[int]) -> set[int]:
+    """Which of `ids` the user's AGENT pulled within OPENED_WINDOW of the call.
+
+    Agent pulls only — the `mcp_` prefix, per NoteUsageEvent's convention —
+    because the question is whether the line moved the session, not whether
+    the operator later browsed to the record.
+    """
+    if not ids or row.created_at is None:
+        return set()
+    async with async_session() as session:
+        found = await session.execute(
+            select(NoteUsageEvent.note_id).distinct().where(
+                NoteUsageEvent.user_id == user_id,
+                NoteUsageEvent.event == PULLED,
+                NoteUsageEvent.source.startswith("mcp_", autoescape=True),
+                NoteUsageEvent.note_id.in_(ids),
+                NoteUsageEvent.created_at >= row.created_at,
+                NoteUsageEvent.created_at <= row.created_at + OPENED_WINDOW,
+            )
+        )
+        return {int(i) for (i,) in found.all()}
+
+
+async def _logged_after(row: RetrievalLog, ids: list[int]) -> set[int]:
+    """Which of `ids` gained a work log after the call. A log lands in its
+    task's embedded document without always touching the task's `updated_at`
+    (a closed task takes no claim stamp), so it is read here."""
+    from grimoire.models.task_log import TaskLog
+
+    if not ids or row.created_at is None:
+        return set()
+    async with async_session() as session:
+        found = await session.execute(
+            select(TaskLog.task_id).distinct().where(
+                TaskLog.task_id.in_(ids), TaskLog.created_at > row.created_at,
+            )
+        )
+        return {int(i) for (i,) in found.all()}
+
+
+def _eligible(user_id: int, source: str, since):
+    """Logged calls of `source` that offered something fresh and that this
+    reviewer has not judged a line of."""
+    judged = exists().where(and_(
+        RetrievalJudgment.retrieval_log_id == RetrievalLog.id,
+        RetrievalJudgment.user_id == user_id,
+    ))
+    return (
+        RetrievalLog.user_id == user_id,
+        RetrievalLog.source == source,
+        RetrievalLog.created_at >= since,
+        RetrievalLog.result_count > 0,
+        RetrievalLog.query.is_not(None),
+        ~judged,
+    )
+
+
+async def menus_to_review(
+    user_id: int, *, source: str = "auto_inject", n: int = 5, days: int = 14,
+    depth: int = DEPTH_DEFAULT,
+) -> dict:
+    """A random sample of unjudged logged calls, each re-run for judging.
+
+    Random, not newest: a review of the latest calls judges one afternoon's
+    work, and the readout would then describe that afternoon.
+    """
+    _check_source(source)
+    n = max(1, min(int(n), SAMPLE_MAX))
+    depth = max(1, min(int(depth), DEPTH_MAX))
+    since = datetime.now(timezone.utc) - timedelta(days=max(1, int(days)))
+    where = _eligible(user_id, source, since)
+    async with async_session() as session:
+        rows = (await session.execute(
+            select(RetrievalLog).where(*where).order_by(func.random()).limit(n)
+        )).scalars().all()
+        remaining = (await session.execute(
+            select(func.count()).select_from(RetrievalLog).where(*where)
+        )).scalar_one()
+
+    menus = []
+    for row in rows:
+        rep: dict = {}
+        lines = await _rerun(user_id, row, depth, report=rep)
+        ids = [ln["record_id"] for ln in lines]
+        opened = await _opened_after(user_id, row, ids)
+        relogged = await _logged_after(row, ids)
+        logged = [int(it["id"]) for it in (row.result_ids or [])]
+        for ln in lines:
+            ln["opened_after"] = ln["record_id"] in opened
+            ln["changed_since_call"] = ln["changed_since_call"] or ln["record_id"] in relogged
+            # Shown fresh then. Inside the budget and not logged means a
+            # repeat the arm withheld, or a ranking the corpus has since moved.
+            ln["logged"] = ln["record_id"] in logged
+        rerun_ids = set(ids)
+        menus.append({
+            "log_id": int(row.id),
+            "created_at": iso(row.created_at),
+            "query": row.query,
+            "budget": row.limit_n,
+            "threshold": row.threshold,
+            "logged_ids": logged,
+            # Logged then, not found now: deleted, edited out of reach, or
+            # outranked since. Judge what is here.
+            "missing": [i for i in logged if i not in rerun_ids],
+            # Written after the call, so never offered to it; left out.
+            "postdated": rep.get("postdated", []),
+            "lines": lines,
+        })
+    return {
+        "source": source,
+        "menus": menus,
+        "remaining": int(remaining),
+        "verdicts": list(VERDICTS),
+        "how_to_judge": HOW_TO_JUDGE,
+    }
+
+
+async def judge_menu(
+    user_id: int, *, log_id: int, verdicts: list[dict],
+) -> dict:
+    """Record a verdict, with its reason, on lines of one logged call.
+
+    Rank, score and budget side are re-derived from a fresh re-run rather than
+    taken from the caller, so a verdict cannot be filed against a position the
+    record does not hold. Judging a line again replaces the earlier verdict.
+    """
+    if not verdicts:
+        raise ValueError("pass at least one verdict")
+    parsed: list[tuple[int, str, str]] = []
+    for v in verdicts:
+        rid = int(v.get("record_id") or 0)
+        verdict = str(v.get("verdict") or "").strip().lower()
+        reason = str(v.get("reason") or "").strip()
+        if verdict not in VERDICTS:
+            raise ValueError(f"verdict must be one of {list(VERDICTS)}, got {verdict!r}")
+        if not reason:
+            raise ValueError(
+                f"record {rid}: a verdict needs its reason — what in the query "
+                f"and the passage decided it. Without one it cannot be re-read."
+            )
+        parsed.append((rid, verdict, reason))
+
+    async with async_session() as session:
+        row = await session.get(RetrievalLog, int(log_id))
+    # Another user's call reads exactly like a missing one: its query is theirs.
+    if row is None or row.user_id != user_id:
+        raise ValueError(f"logged call {log_id} not found")
+    _check_source(row.source)
+
+    lines = {ln["record_id"]: ln for ln in await _rerun(user_id, row, DEPTH_MAX)}
+    unknown = [rid for rid, _v, _r in parsed if rid not in lines]
+    if unknown:
+        raise ValueError(
+            f"records {unknown} are not in this call's re-run to depth "
+            f"{DEPTH_MAX} — judge the lines menus_to_review returned"
+        )
+    opened = await _opened_after(user_id, row, [rid for rid, _v, _r in parsed])
+
+    async with async_session() as session:
+        existing = {
+            int(j.record_id): j
+            for j in (await session.execute(
+                select(RetrievalJudgment).where(
+                    RetrievalJudgment.retrieval_log_id == row.id,
+                    RetrievalJudgment.user_id == user_id,
+                    RetrievalJudgment.record_id.in_([rid for rid, _v, _r in parsed]),
+                )
+            )).scalars().all()
+        }
+        for rid, verdict, reason in parsed:
+            ln = lines[rid]
+            j = existing.get(rid) or RetrievalJudgment(
+                user_id=user_id, retrieval_log_id=int(row.id), record_id=rid,
+            )
+            j.source = row.source
+            j.query = row.query
+            j.rank = ln["rank"]
+            j.score = ln["score"]
+            j.within_budget = ln["within_budget"]
+            j.verdict = verdict
+            j.reason = reason
+            j.opened_after = rid in opened
+            session.add(j)
+        await session.commit()
+    return {
+        "log_id": int(row.id),
+        "recorded": len(parsed),
+        "replaced": len(existing),
+    }
+
+
+async def judged_block(user_id: int | None, since) -> dict:
+    """`retrieval_telemetry`'s judged block: verdicts by rank, per source.
+
+    Counts, not rates, beside each other per rank — the sample is small by
+    nature and a percentage of seven reads as more than it is. `on_point_unopened`
+    is the number the open rate could never show: lines that related and were
+    left closed, which is the passage doing its job or the reader missing it.
+
+    Raises on a failed read; the caller guards it.
+    """
+    async with async_session() as session:
+        rows = (await session.execute(
+            select(
+                RetrievalJudgment.source,
+                RetrievalJudgment.rank,
+                RetrievalJudgment.within_budget,
+                RetrievalJudgment.verdict,
+                RetrievalJudgment.opened_after,
+                func.count().label("n"),
+            )
+            .where(
+                RetrievalJudgment.user_id == user_id,
+                RetrievalJudgment.created_at >= since,
+            )
+            .group_by(
+                RetrievalJudgment.source, RetrievalJudgment.rank,
+                RetrievalJudgment.within_budget, RetrievalJudgment.verdict,
+                RetrievalJudgment.opened_after,
+            )
+        )).all()
+        calls = dict((await session.execute(
+            select(
+                RetrievalJudgment.source,
+                func.count(func.distinct(RetrievalJudgment.retrieval_log_id)),
+            )
+            .where(
+                RetrievalJudgment.user_id == user_id,
+                RetrievalJudgment.created_at >= since,
+            )
+            .group_by(RetrievalJudgment.source)
+        )).all())
+
+    def empty() -> dict:
+        return {v: 0 for v in VERDICTS} | {"opened": 0}
+
+    out: dict = {}
+    for source, rank, within, verdict, opened, n in rows:
+        block = out.setdefault(source, {
+            "judged_calls": int(calls.get(source, 0)),
+            "judged_lines": 0,
+            "within_budget": empty(),
+            "beyond_budget": empty(),
+            "by_rank": {},
+            "on_point_unopened": 0,
+        })
+        n = int(n)
+        block["judged_lines"] += n
+        side = block["within_budget" if within else "beyond_budget"]
+        at = block["by_rank"].setdefault(int(rank), empty())
+        for bucket in (side, at):
+            if verdict in VERDICTS:
+                bucket[verdict] += n
+            if opened:
+                bucket["opened"] += n
+        if verdict == VERDICTS[0] and not opened:
+            block["on_point_unopened"] += n
+    for block in out.values():
+        block["by_rank"] = [
+            {"rank": r, **c} for r, c in sorted(block["by_rank"].items())
+        ]
+    return out