A review pass judges whether injected lines related (#4772) #196

Merged
bvandeusen merged 1 commits from dev into main 2026-10-03 18:17:49 -04:00
Owner

#4772 "Injected notes are never judged — "opened or not" can't tell an unrelated line from one whose passage was enough, so the open rate can't settle the budget"

  • menus_to_review / judge_menu MCP tools: re-run a sample of logged auto-inject calls past the budget and record a verdict (on_point / adjacent / unrelated, with a reason) per line.
  • retrieval_judgments table (migration 0115).
  • retrieval_telemetry gains a judged block: verdicts by rank, within vs beyond the budget, on-point-but-unopened.
  • The surfaced_never_pulled warning no longer prescribes a title fix; it points at the review.
  • missed-retrieval skill text; plugin version 2026.10.03.1934.

Dev CI run 7964 green.

🤖 Generated with Claude Code

#4772 "Injected notes are never judged — "opened or not" can't tell an unrelated line from one whose passage was enough, so the open rate can't settle the budget" - `menus_to_review` / `judge_menu` MCP tools: re-run a sample of logged auto-inject calls past the budget and record a verdict (on_point / adjacent / unrelated, with a reason) per line. - `retrieval_judgments` table (migration 0115). - `retrieval_telemetry` gains a `judged` block: verdicts by rank, within vs beyond the budget, on-point-but-unopened. - The `surfaced_never_pulled` warning no longer prescribes a title fix; it points at the review. - missed-retrieval skill text; plugin version 2026.10.03.1934. Dev CI run 7964 green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
bvandeusen added 1 commit 2026-10-03 18:17:44 -04:00
feat(retrieval): a review pass judges whether injected lines related — menus_to_review, judge_menu and a judged readout (#4772)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 1m1s
CI & Build / Python tests (push) Successful in 1m55s
CI & Build / Build & push image (push) Successful in 44s
6598c7fa85
An open rate cannot say whether a menu line related: every line carries its
matched passage (#4364), so "not opened" covers unrelated, enough as shown,
and already in context. #4772 "Injected notes are never judged".

- retrieval_judgments (0115): a reviewer verdict per line of a logged call,
  on_point / adjacent / unrelated, with its reason, rank, budget side and
  whether the agent opened it within the hour.
- menus_to_review re-runs a random sample of unjudged auto_inject calls with
  the arm's own parameters, past its budget, passage on every line.
  judge_menu records verdicts, re-deriving rank from a fresh re-run.
- retrieval_telemetry gains a judged block (by rank, within/beyond budget,
  on_point_unopened). surfaced_never_pulled stops blaming titles.
- missed-retrieval guidance names the review before a budget move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bvandeusen merged commit c50c1beb83 into main 2026-10-03 18:17:49 -04:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#196