7 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	8be555d6dd	feat(telemetry): tell a ranker decline from a repeat before the observation window opens (#3497) ... Making the rule arms log every call exposed a second ambiguity in the same row. `result_count == 0` is two unrelated events wearing one number: - the ranker found nothing above the bar — the only evidence a threshold is set too high; and - the ranker found only what this session had already been shown — which says nothing whatever about the bar. A long session excludes its way into the second, so the arm reads worse the longer it runs correctly. Rows written now carry the ambiguity permanently, which is why this lands before any watch period rather than after. `retrieval_logs.suppressed_count` (0095, nullable) holds what the caller dropped as already-shown. Both rule arms report it; they filter in Python and always know. The note arms pass exclusions INTO semantic_search_notes and never see what was dropped, so they store NULL. THE NULL IS LOAD-BEARING. It means "not measured here", and the readout renders it as `suppression: null` rather than a zeroed dict. Defaulting to 0 would let an unmeasured surface read as a perfectly clean one — the same substitution of an artifact for a measurement that #3311 made. No backfill, for the same reason: existing rows genuinely do not know. `retrieval_telemetry`'s `sources` gains `suppression` with `measured_calls`, `calls_with_suppression` and `zero_because_already_shown`; subtract the last from `zero_result_calls` for the true ranker declines. The MCP tool docstring says to read the two together and warns against reading the null as a zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 21:12:38 -04:00
bvandeusenandClaude Opus 5	154a5de13e	fix(telemetry): both rule arms logged only their hits, so the clear-rate could only read 100% (#3497) ... `write_path_rule` reported `zero_result_calls: 0` and `cleared_threshold: 133/133` — a perfect record no other surface comes near (`write_path` 421 zeroes of 613, `reuse_slot` 124/199, `auto_inject` 114/326). #3311 read that as a measurement and milestone 333 was scoped on it. It was an artifact. Both arms called `record_retrieval` inside a guard on having results — the write-path arm behind `if fresh:`, the pre-tool arm below `if not fresh: return out` — so a call that found nothing wrote no row. The statistic was a fact about the shape of the code, true at any threshold whatsoever. The call log moves out of the guard in both arms. The surfacing log stays in it: nothing was shown, so no surfacing occurred. `results=fresh` is kept deliberately — the note arms pass exclusions into `semantic_search_notes`, so what they log is already post-exclusion, and logging `hits` here would make this row mean something other than every other row in the same readout. The defect bites hardest on the pre-tool arm, which fires on every Bash call: with no rows at all, a ranker that declined is indistinguishable from a hook that never fired — the silent failure the arm exists to stop. Tests cover both arms behaviourally (found nothing; found only what the session already held; searched nothing at all, which must stay silent) plus a structural guard, because this was one level of indentation and it appeared independently in two places. #3311 and the `rule_usage` docstring corrected rather than quietly rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 06:57:13 -04:00
bvandeusenandClaude Opus 5	2ee24b9d2b	feat(rules): rules before tools — a PreToolUse arm keyed on the action (#3476) ... The only just-in-time rule surface was registered on `Write|Edit` and queried with `code or path`, so a rule could be retrieved at the moment of a code write and nowhere else. Every rule about which tool to reach for — don't curl the forge, don't stand up a stack, don't run the suite locally, don't branch — was unreachable exactly when it mattered, and residency in the always-on preload was the only surface it had. That is the pressure that grew the resident set to 31 against #3089's ceiling of ~23; it was never a judgment anybody made. A reflex generates no query, so an instruction to check the rules cannot catch one. A mechanical trigger can: the tool call IS the query, and a reflex has to become a tool call before it can do anything. `build_tool_rule_hint` is deliberately tool-agnostic — a name and a string — so widening the matcher later is a hooks.json edit with no server change. The hook starts on Bash, which is where the action reflexes live. The two pre-tool arms share ONE session ledger of already-named rules (`<state>/<sid>.rules.ids`). Two ledgers would mean a rule named by one arm gets re-offered by the other, and the hint that fires most often is exactly the one that must not repeat itself. A test asserts both scripts build the same path, and another checks the shell hook and the Python route agree on every query-arg name (rule 33) — a rename there fails silently, looking like a surface that never finds anything rather than a broken one. Deliberately silent on outage, unlike the prior-art hook: a write is occasional, a Bash call is not, and an outage line before every command is what gets a channel muted. `tier="conditional"` matches the write arm and is the transition point — an always-on rule is already resident, so re-tier one and it starts arriving here instead of in every session's preamble. `pre_tool_rule` joins RANKED_SOURCES: this arm chose what it showed, so a pull can settle whether the choice landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-02 23:31:22 -04:00
bvandeusenandClaude Opus 5	8b9b3a1d9b	feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473) ... The ranked rule arm became measurable in M333. The preload did not — and that is the surface whose value is actually in question. `list_always_on_rules`, the SessionStart block and every `rules_payload` caller handed rules over wholesale and emitted nothing, so the resident set's token cost was certain and its usefulness could not be tested even in principle. Bulk deliveries now record as AMBIENT, beside the ranked count and never inside pull-through. Folding them in would mean growing the always-on set depressed the arm's measured precision and trimming it flattered the arm, neither for any reason to do with the arm. `RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there is one ranked rule source and this change adds seven bulk ones, so naming the rare half makes a forgotten surface default to ambient — under-counting it — rather than padding the denominator with surfacings nobody chose. Two lookalike call sites are deliberately left silent, with a test to keep them that way: the write-path etag arm and `rules_etag_for` read the rules to build or compare a MARKER and show nobody anything. No migration — `event` and `source` are plain Text with no CHECK (rule 36 does not apply). Snippet #2858 updated to the new `rules_payload` contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-02 23:06:40 -04:00
bvandeusenandClaude Opus 5	238510080e	feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318) ... Milestone 333 step 4 — the split #2223 made one surface down, now made for the third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT. THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two facts hold on any install, including one with six rules and no telemetry: - The eligible corpus is tiny — conditional rules only, a handful to a few dozen against thousands of notes. A top-k over forty candidates always returns something, so "the best match cleared the bar" stops meaning "a good match exists". A bar calibrated for best-of-thousands is cleared by best-of-forty as arithmetic, not relevance. - Rules are short imperative technical English, far more homogeneous than note prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it; a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely inherited, it sits below where this corpus's noise lives. 0.72 errs deliberately toward silence on an asymmetry that is also structural: this hint fires on EVERY write. A missed rule is recoverable — it is still in Scribe and the agent can search it. A hint that cries wolf is not: it teaches the reader to skip the whole block, and the true positives go with it. The arm's own comment already said "noise on a hint that fires on every write is how a hint gets ignored". Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting the relationship — which would silently reinstate #3311 — does not. RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2 means the second line is almost always the second-best noise wearing the same confident framing as the first; halving k halves that regardless of the bar. It stays a constant because it is a decision about how loud one hint may be, not a per-install tuning question — and a knob nobody turns only adds a way to misconfigure the surface. Reachable from Settings, no restart (rule 25), with copy that says which way to move it and points at retrieval_telemetry's rule pull-through — which step 3 made readable — to tell "arriving unread" from "never arrived". Every config stand-in in the suite gained the key, not just the one that noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so a missing key raises inside its fail-open except and turns the arm into a silent no-op — indistinguishable from it running and finding nothing. That is the same vacuous-pass shape that bit step 2, one layer down (rule 33). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 18:05:00 -04:00
bvandeusenandClaude Opus 5	70761b16d9	test(telemetry): the rule-arm fixture never reached the arm — it returned at the guard (#3316) ... The product code was right; the test was wrong, and wrong in a way that made two assertions fail and two others pass vacuously. `build_write_path_hint` returns early when a write matched nothing at all — no staleness, no synced record, no prior-art menu, no shape signal. The rule arm sits deliberately on the FAR side of that guard, because it runs a semantic search and hoisting it would mean an embedding query on every write in the session. My fixture stubbed every other arm to empty, so it hit the early return and the rule arm never ran: `record_rule_surfaced` was called zero times, and "the recorder was not called" is also what two of the four tests were asserting for their own reasons. The fixture now supplies one prior-art hit — 0.72 against a 0.6 threshold, so it clears the band and the top_k slice — with a comment saying the hit is the arm's precondition rather than scenery. And the gate got its own test, because the fixture now depends on it: a write matching nothing must NOT reach the arm. Without that, a future change to the guard would make every assertion in this file pass without exercising anything. #3311 is explicit that the gate stays until the arm's precision is fixed, so the test says to go read that issue rather than update the assertion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 17:17:59 -04:00
bvandeusenandClaude Opus 5	8f7f447fda	feat(telemetry): the rule arm records what it showed, and get_rule records the read (#3316) ... Milestone 333 step 2. Step 1 built the table; a counter nobody calls reads zero and looks exactly like a surface nobody uses, which is #2663's shape. SURFACED — the standing-rule arm in build_write_path_hint, beside the record_retrieval it already made. Two tables, and the split is not arbitrary: retrieval_logs is one row per CALL keyed on the score distribution a threshold is tuned from; rule_usage_events is one row per RULE per event, the grain "was this hint ever acted on" needs and the grain a JSONB result_ids array cannot be indexed at. The comment there said rule ids had nowhere to go — that note_usage_events remaps ids on restore, so a rule id would return attached to whatever note took that number. Still true of the NOTE table, and precisely why step 1 built its own. Rewritten to say the gap is closed rather than leaving a stale rationale that would have someone re-derive the same dead end. Records `fresh`, i.e. AFTER exclude_rule_ids. A rule the session already holds was considered and not shown; counting it would inflate the denominator with claims the agent never saw, and the ratio would then fall for a reason that has nothing to do with whether hints land. PULLED — two doors, both after their access check so a refused read is not a pull. mcp_get_rule is the one that matters: the arm's own message ends "Read it with get_rule(N)", so that call is the exact action a landed hint produces. rest_rule carries the other prefix, and the prefix is load-bearing — "is this rule dead weight?" is served by any pull, "did that injected hint land?" by agent pulls only. NOT a pull: rule_history. It loads the rule for its title and its own output says "The current wording is on the rule itself — get_rule(N)", so counting it would credit a read of the history as a read of the rule and double-count anyone who then follows that pointer. list_always_on_rules and enter_project are likewise bulk resident loads, not somebody choosing to open one record. tests/test_rule_usage_wiring.py is cross-cutting on purpose: the surfaced end is in plugin_context, the pull end in two other modules, and "both ends meet" is a property no module-shaped file asserts. It covers the exclusion boundary, that a failing recorder cannot break the write, that a refused read records nothing, and two completeness guards — every door records, and the bulk loaders still do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 17:15:03 -04:00