1430 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusen	6da39e0298	Merge pull request 'The prompt-boundary rule arm, and a slot a preference cannot lose (#3852, #3894)' (#151) from dev into main	2026-09-11 11:11:59 -04:00
bvandeusenandClaude Opus 5	d5f96563fd	feat(rules): a slot a preference cannot lose (#3894) ... Milestone 399 step 4. A rule and a preference are not equally served by one ranking, because their losses are not equal: - a RULE crowded out at the prompt boundary still fires at an act arm. A push reaches pre_tool_rule, a write reaches write_path_rule. The prompt hit is a preview of a second chance. - a PREFERENCE about how to answer has no second chance. The response IS the act, so crowded out there it is never delivered at all. A straight ranking therefore favours the record whose loss is recoverable over the one whose loss is total, and does it INVISIBLY: the rule that won is a legitimate hit, the telemetry reads healthy, and the only symptom is a preference that quietly never arrives. reuse_slot exists for the same shape one corpus over (#2463). `semantic_search_rules` gains a `kind` filter, so the slot's query can only answer with what the slot is for. Verifying afterwards would be weaker — an unfiltered search that happened to return a rule would spend the slot on it, and that line would be indistinguishable from one that earned its place. THE SLOT BUYS POSITION, NOT A LOWER BAR, matching reuse_slot. A weak preference cannot buy it, so silence stays the default. The task asked for a separate threshold; I did not add one, and the reason is that the worry behind it — reading a miss rate as a fact about preferences — is answered by `preference_slot` being its own logged source, where best_available_id names which preference was refused. A knob added on a guess is a way to misconfigure the surface; a bar moved on evidence is an argument. The evidence arrives on its own now. IT EXTENDS, IT NEVER DISPLACES — and here it parts from reuse_slot, which evicts its menu's weakest hit. A displaced hit sits in prompt_rule's retrieval_logs row while never being surfaced, so that source's two tables stop agreeing and #3668's identity breaks for a reason nothing in the data explains. Milestone #379 is what losing that identity costs: five steps planned against two counters disagreeing, not a write path dropping rows. One extra line in a rare case is the cheaper price. It also runs BEFORE the bail-out. An empty general result is not proof no preference qualifies: that search overfetches by distance then collapses, so a preference ranked below the window is invisible to it while a kind-filtered query finds it at once. Bailing first would make the slot dead in exactly the corpus it exists for. One existing assertion repinned from a bare call_count to a per-source filter: the slot logs its own query on the same call, and a count would pin the number of arms rather than the property — going red the next time one is added, which is rule 167's false alarm about the thing it protects. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 08:44:24 -04:00
bvandeusenandClaude Opus 5	44e0b0541f	feat(rules): rules retrieve against the operator's message (#3852) ... The third rule arm, and the one the other two cannot reach. `write_path_rule` is keyed on code, `pre_tool_rule` on a command — both things the session is about to DO. A rule that governs what to SAY has no such trigger: extract intent from loose phrasing, raise a conflict before acting, hand off an action with its reason, end a finding with an offer all bind on a RESPONSE, and no tool call precedes one. The operator's message is the only query that exists before a response is composed. That hook searched notes alone, so no rule had ever been retrieved against a thing the operator actually said — and residency was the only surface those rules had, which is what milestone 394 removes. A SEPARATE FUNCTION, not a branch in build_autoinject_hint, because of its early returns. That arm bails when auto-inject is disabled, when the query is blank, when nothing clears the note bar — every one a statement about NOTES. Folded in, an operator who turned the awareness menu off would silently lose their rules, a coupling with no symptom since both look like a quiet hook. Two functions, two sets of gates, composed in the route. Guarded as "the rule arm never asks the notes arm's config", which is the structural fact. Joins _ARMS rather than getting its own test file. #3497's history is that the pre-tool arm inherited a defect from its sibling by being MODELLED on it instead of sharing with it, and a third arm modelled on two is two chances to repeat that. Repeat rendering, fresh-only counting, log-before-bailout, the kind register and the two-recorders identity are properties of every arm or of none. The bar is INHERITED and says so. 0.72 was tuned against code and commands; prose is a different query shape against the same documents, and triggers are written in the vocabulary of the moment — which for most rules is act vocabulary. Starting at the only number with evidence behind it and logging every call from the first deploy is what makes it settleable; guessing lower would put an unmeasured bar in front of a corpus that binds. k=3, anchored on this hook's own budget rather than the act arms'. RULEHINT_LIMIT is 1 because that arm fires before every Bash call; this one fires once per turn, beside a notes menu already spending three slots. And a prompt genuinely contains more than one act — "merge to main and then start on X" is two — where a command is one thing. `prompt_rule` added to RANKED_SOURCES: a ranker picked it, and a ranked source missing from that tuple is silently counted as bulk delivery and drops out of the pull-through denominator. The hook reads and writes the SHARED rule ledger under scribe-priorart, not a private one — one session keeps one list, aged (#3751), so a rule named here is not re-announced before the next Bash call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 07:56:19 -04:00
bvandeusen	5f77fc29c1	Merge pull request 'Preferences — a rule kind that Scribe keeps up to date (#3849 steps 1–3)' (#150) from dev into main	2026-09-10 23:34:16 -04:00
bvandeusenandClaude Opus 5	8406871085	fix(plugin): the instruction surfaces still said every rule binds (#3849) ... Step 3 shipped a line a session can receive — "Preference that may apply here …" — into surfaces that told it, in the most authoritative voice it has, that anything arriving in that shape is binding. That is the confusion milestone 399 exists to prevent, arriving through the one channel a session has least reason to doubt. Silent in both directions, which is why it could not wait for step 6. A session treating a preference as a rule refuses to proceed over something the operator merely preferred; and it loses the whole reason preferences exist, which is that they are brought up to date rather than obeyed. Three surfaces, each to its own budget: - SKILL.md gets the full account: kind decides force, the injected line names which in its opening words, and a preference is the one record a session keeps current itself (update_preference, with what taught the change). - scribe_static_context.md gets six lines — enough to tell the kinds apart and to say a preference is yours to update. - _INSTRUCTIONS gets four words. It is a MAP at 1978 of its 2000-char budget (#2562), and the detail belongs in the surfaces above and in the tool docstrings, which is what that budget exists to force. Guarded so it cannot drift back: a surface that claims rules bind must name the kind that does not. Pinned on the CLAIM rather than the word "bind", because a bare substring also matches bind_repo, list_repo_bindings and server.py's DNS-rebinding comment — a guard that would one day fail a skill about repo binding is rule 167's named failure, raising a false alarm about the very thing it protects. Falsified against all three surfaces losing the mention. Plugin version minted: the cache refreshes only on a version bump (#2209), so a skill edit without one reaches no installed plugin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-10 23:20:04 -04:00
bvandeusenandClaude Opus 5	26e0dff706	feat(rules): a preference does not speak in a rule's voice (#3849 step 3) ... Two independent axes on one hint line. `kind` decides the head, `seen` decides the tail, and neither reads the other — which is what let a second kind arrive without reopening #3750's repeat question. Whether a record is already on the exclusion ledger has nothing to do with how much force it carries, so the seen branch is shared verbatim. The noun carries the whole visual difference, deliberately. A reader skimming an injected block gets one word to place the register, so the word that moves is the one naming force: "Standing rule" / "Preference". Everything structural after it is identical, so the kinds read as one set rather than two formats. Force is asserted in exactly one other place, and that moves too. A rule's line says to read it BEFORE DECIDING IT DOES NOT APPLY, because dismissing a rule unread is how the thing it prevents happens. A preference makes no such claim: it says where to find HOW THIS HAS BEEN DONE BEFORE, and following it buys consistency rather than correctness. Guarded on both places at once. Pinning the noun alone would pass a line reading "Preference … before deciding it does not apply" — label swapped, instruction kept — which is worse than not distinguishing them, because it looks handled. And a guard on the independence claim itself, exercising all four combinations: the way this breaks silently is a seen branch that grows a kind test, leaving one combination rendered by nobody's intention. Noted, not fixed: plugin/skills/using-scribe/SKILL.md still says "Standing rules are binding" with no room for a kind that does not. That surface is step 6's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-10 22:04:14 -04:00
bvandeusenandClaude Opus 5	89d16d89a9	feat(rules): a preference updates without asking, and says what taught it (#3849 step 2) ... The write path, and the step where a preference stops being a relabelled rule. `create_preference` / `update_preference` on the MCP surface, plus `kind` on update_rule and both HTTP doors. SEPARATE TOOLS, NOT A `kind=` ARGUMENT. create_rule's docstring IS the approval gate (#3557): propose, offer three answers, wait. That is right for a rule — the person it binds should have agreed. A preference inverts it, and one reached through create_rule would be read through that prose, so the caller would hesitate over exactly the act this kind exists to make routine. Two doors, two contracts, one table. Reads stay shared: a preference IS a rule row, and "what governs this" wants both. Two required fields, each buying something: - `when_to_apply`, because the trigger is two-thirds of the embedded document. Without one the record is written, stored, and silently never delivered — indistinguishable from one nobody wrote. - `arose_from_id`, the price of the ungated write. A corpus that drifts with no record of what taught each change cannot be audited, and the operator's veto over drift is worth exactly as much as their ability to read why it happened. The near-duplicate gate is what lets this corpus be written freely and stay small: the second preference about a thing updates the first. It is title-scoped and kind-blind, so it also catches a preference restating a rule that already binds. The asymmetry is guarded as two PRESENCE facts — the rule door still asks, the preference door still says write it — never as an absence. An absence check passes against a docstring that was deleted or rewritten into something else, which is snippet #3352's warning and would read as coverage here while proving nothing. `_plain_detail` moved to tests/helpers on its second copy, per that module's own reason for existing (#2825). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-10 21:59:23 -04:00
bvandeusenandClaude Opus 5	4aae4973f7	fix(tests): the backup stand-ins predate kind (#3849 step 1) ... Two unit tests build a rule with SimpleNamespace rather than the model, so adding a column broke them — the fixture has no `kind` for `_rule_rows` to read. Fixture-only; the export itself was already right, which the existing column-coverage guard confirmed by passing. Adds the guard that coverage check cannot make. `_stand_in` walks `__table__.columns` and proves the KEY is emitted; it cannot prove the VALUE survives. A preference exported as a rule is a silent failure — the restored rule reads fine and simply binds when it was only ever a preference — and a fixture carrying the default would pass against a `_rule_rows` that dropped the field and let the importer's `or "rule"` refill it. So the new test asserts on `preference`, the one value that cannot be reconstructed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-10 21:31:16 -04:00
bvandeusenandClaude Opus 5	c63172272d	feat(rules): a preference is a rule that does not bind (#3849 step 1) ... Adds `kind` to rules — `rule` binds, `preference` is how the operator wants work done. One column, because the two differ in exactly one dimension and everything else a preference needs already lives on `rules`: a trigger column, a trigger-dominated embedding document, ownership-scoped search, three retrieval arms with telemetry, typed relations, and versioning. Defaults to `rule`, so nothing changes force on upgrade — 0088's argument for `tier`, unchanged. `rule_versions` gets the column too, and that half is not bookkeeping. `record_if_changed` decides whether an edit deserves a snapshot by comparing the fields a version carries, so a field absent from SNAPSHOT_FIELDS is a field whose change records no history at all. Without it, turning a rule into a preference — the moment something stops binding, and the single most consequential edit either kind can undergo — would leave the history silent. Backup carries it through all four seams. A missed one would have restored every preference as a rule, quietly. Guarded on real Postgres in three halves: a preference writes, a typo is refused (without which every other assertion would pass against a table whose CHECK had been dropped), and a row written with no kind reads back as `rule` — the migration's whole safety claim, asserted rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-10 21:14:29 -04:00
bvandeusen	d8017f9b96	Merge pull request 'A surfaced rule stops going permanently quiet (#3750, #3751, #3752)' (#149) from dev into main	2026-09-09 22:44:28 -04:00
bvandeusenandClaude Opus 5	c1aa1d8e92	feat(plugin): rule exclusions age, so salience decays without a context event (#3751) ... #3749 clears the ledger when an EVENT destroys context — a compaction, a /clear. This is the case with no event at all: a long session where a rule was named two hundred turns ago and has simply fallen out of attention. It is #3702's argument at the tier level — present in context and salient at the moment are different properties — applied to time instead of to tier. FORMAT: `id<TAB>epoch`, per entry. Not a whole-file mtime: that is one line of shell and wrong in exactly the session that needs it, since a single recent write keeps every stale id alive, and the ids that go stale first come from the rules that fire most. Not a turn counter, though it would be the truer model — an idle session does not forget. A hook has no turn number without keeping its own, which is a second piece of session state to write, read, clear on compaction and get wrong. Wall time costs a `date` call. The failure it accepts is a session left idle over lunch treating its rules as forgotten, worth one extra full line per rule and nothing else. TTL 2700s (45 minutes), reasoned rather than picked (rule 32). About one working stretch on a single task: long enough that a rule does not re-announce itself while you are still doing the thing it governs, short enough that a multi-hour session gets a refresh rather than one 9am mention. It leans short because since #3750 being wrong on the short side is the cheaper error — an expired entry costs one full line instead of one short one, and the exclusion re-arms the moment it is spent. There is no data on this yet; #3807's near-miss listing is what should revise it. BOTH READERS THROUGH ONE HELPER, in scribe_defs.sh. The two hooks share one ledger so a rule named by one arm is not re-offered by the other; a format only one of them understood would break that on the first read. The flat `tr '\n' ','` read would now send `156<TAB>1789002860` as an exclude id — verified, which is why this is not a per-hook edit. THE LAST ENTRY FOR AN ID WINS. The file is append-only, so a rule that ages out, is surfaced fresh and is appended again has two lines. Reading the first leaves it permanently expired, and it then re-announces itself on every call for the rest of the session — the mechanism meant to quieten things becoming the loudest thing in the hint. A BARE ID IS LIVE. That is the pre-#3751 format, and every session in flight when this ships has a ledger full of them. Reading unknown as EXPIRED would make all of those sessions re-announce every rule they had already been told, at once — the exact noise this prevents, delivered by the feature on the day it ships. Unknown means "not measured", never "old", the same discipline the nullable retrieval_logs columns use. GUARDS (tests/test_rule_ledger_ageing.py, real shell, no credentials) - old ages out AND recent survives, in ONE assertion (rule 167): either half alone passes against a broken helper — "old is gone" passes against one returning nothing, "recent survives" passes against the flat read this replaces, i.e. against the defect itself. - the ping-pong case, which is the one that costs the most to get wrong. - a bare id is live, including beside a stale stamped one. - missing/empty ledger excludes nothing. - an id repeated in the ledger appears once, with no empty list element. - structural: neither hook reads the rule ledger flat again — pinned on the flat-read SHAPE, so a rename of the helper is not a failure and a hook that ages correctly some other way is not either. The TTL's VALUE is deliberately not asserted. The tests read the constant out of the shell and assert the property around it, so a later tuning change stays a tuning change instead of a red build. Dropped a boundary test (`ttl` vs `ttl + 1`) before committing: racy by construction, since a ledger written at T is read at T+n and the two cases swap. A one-second distinction on a 45-minute window is also not observable behaviour, so it pinned a flake rather than a property. Plugin version minted to 2026.09.10.0221 — this is entirely hook-side, so without the bump the cache would never pick it up and the merge would ship nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 22:21:32 -04:00
bvandeusenandClaude Opus 5	c4908f093f	feat(rules): a repeat is referenced, not withheld (#3750, #3752) ... Both arms used to drop a hit already on the session's exclusion ledger and emit nothing. That is correct only while the session still HOLDS what it was told, and a compaction breaks exactly that: the earlier injection is summarized away while the id stays on the ledger, leaving the rule absent from context AND unreachable for the rest of the session. #3749 closed the compaction half by clearing the ledger; this closes the ordinary half, where a session simply stops holding a line it read an hour ago. Only ONE CLAUSE of the existing line is false on a repeat — the claim that the rule is not in the session's loaded set — so only that clause changes. The fresh line is byte-identical to what it was. Both tails now come from one `_rule_hint_line`. The arms phrase their heads differently on purpose; everything after must not differ, and #3497's history is that the pre-tool arm inherited a defect by being modelled on its sibling rather than sharing with it. THE BUDGET DECISION, recorded at RULEHINT_LIMIT. A repeat competes for the single slot on rank alone: nothing is fetched behind it, and it never rides alongside as a second line. Promoting a fresh rule past a better-ranked repeat would reinstate the withholding one rank deeper, and a second line is the one thing the limit exists to forbid. The consequence is deliberate — a rule that keeps ranking first for a recurring situation keeps being referenced, and its decay belongs to exclusion ageing (#3751), not to a first-place rule being demoted for having won before. THE TELEMETRY, decided before shipping rather than after a number moved (#3752): nothing changes. A reference is a RENDERING decision, not a retrieval outcome. `results` stays `fresh`, `suppressed` stays len(hits) - len(fresh), and `record_rule_surfaced` still counts only what the arm freshly chose. This matters more than it reads: the naive implementation drops the `fresh` filter and takes suppressed_count to zero everywhere — and #3739's near-miss fix identifies repeat-caused zeros by `suppressed_count > 0`, so the contamination corrected on 2026-09-08 would return by a different route, in the same field, with the fix still in the code and no longer working. A test asserts the counters as unmoved, because "nothing changed" is only worth something if it is checkable. Also corrects two comments that outlived #3702 — both arms still claimed CONDITIONAL ONLY while the module-level note above RULEHINT_LIMIT said the opposite, in the exact code this change edits. GUARDS - an already-held hit produces a rule line at all (the regression), asserted on `get_rule(<id>)` rather than a truthy context: the write-path arm fills its context from four other sources, so truthiness passes under the OLD behaviour and pins nothing there. - the two tails are distinguishable, each excludes the other, and neither injects the rule statement — the budget claim, both branches. - the counters are unmoved, per #3752. - test_a_rule_the_session_already_holds_is_not_re_offered asserted the old contract (`"161" not in context`). Repinned rather than deleted: the telemetry half of what it protected still holds. Repinned the #3497 log-placement guard on structure (rule 167). It read `body.index("if not fresh:")` — a local variable NAME, not the property. This change renames that guard to `if not hits:`, so the old assertion would have raised ValueError and reported #3497 as back while the arm was entirely correct. Now walks the AST for the first early return after the search and asserts the call row is written before it. Falsified both ways: it fails on the #3497 mutation, and it refuses to pass when no early return exists at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 21:48:50 -04:00
bvandeusen	a9fa8bddb6	Merge pull request 'Record WHAT the bar turned away, not only how close it came (#3807)' (#148) from dev into main	2026-09-09 21:34:56 -04:00
bvandeusenandClaude Opus 5	d5ac8408f6	feat(telemetry): record WHAT the bar turned away, not only how close it came (#3807) ... #3670 added `best_available_score` so a threshold could be judged from its rejections. It records how CLOSE the bar came to firing and not WHAT it refused, and that is the half a decision actually needs. Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 — about 117 declines a day within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to roughly 139: six-fold, on a surface that runs before every Bash call. The percentile says the mass is there. Nothing said whether it was worth showing. NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected rule line already carries title and trigger, so a session can comply without ever calling get_rule, and rule pull-through understates usefulness by construction. Reading the rejected records can — and `result_ids` holds only what was RETURNED, so on a zero-result call the near-missed record had no name at all. So the id, from the SAME ranked candidate as the score. Both searches unpack `best` once and read both fields off it, because splitting that into two expressions is exactly how a later edit pairs a score with its neighbour's id — and a score attached to the wrong record is worse than no id, since it invites judging the wrong one and concluding the bar is fine. write_path withholds the id on the same condition it withholds the score (#3739): a surviving id beside a null score names a record without saying what it scored, the pair disagreeing in the other direction. THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled, and a reader tuning a bar needs to go and read the records. Opt-in via `near_miss_samples` (0-20, default 0) so the ordinary readout keeps its size, and deliberately NOT a window function: this module's one production outage was a grouped query Postgres rejected, swallowed by the broad except, every counter reading zero while the mocked tests passed (#2663). One flat ordered query, overfetched, bucketed in Python — the shape that lesson prescribes. Migration 0097, nullable and unbackfilled. Not a foreign key: the table spans record types and `source` says which, exactly as result_ids works. The integration guard pins the listing as PER SOURCE. A global LIMIT would let a noisy source eat the whole quota and leave the surface being tuned showing nothing — which reads as "nothing was close", the misreading this milestone has spent itself correcting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 21:24:32 -04:00
bvandeusen	cd910e32a0	Merge pull request 'A search that never ran is not a decline' (#147) from dev into main	2026-09-09 13:36:43 -04:00
bvandeusenandClaude Opus 5	623464323e	fix(telemetry): a search that never ran is not a decline (#3765) ... `best_available_score` was added by #3670 so a bar could be judged from what it rejected, and it arrived null on four unrelated causes: the corpus offered nothing, the query was empty, the embedder was down, or the DATABASE QUERY FAILED. Only the first is a measurement. The fourth is the #2663 shape — a swallowed failure rendering as a clean zero — inside the field added to fix an instance of the #2663 shape. Found while trying to explain why reuse_slot returned nothing on 45 of 45 calls, and auto_inject on 153 of 161. That investigation is still open; what it established first is that the readout could not answer it. THE FIX IS NOT A NEW COLUMN. A call that never searched writes no row, so every remaining null means one thing: searched, and nothing came close. That is the convention the pre-tool arm already follows for a blank command — "a row here would report a call that never happened and drag the clear-rate down with phantom declines" — extended from the case a caller can see in advance to the ones only the search knows about. Both searches stamp `report["searched"]` FALSE before anything can return and True only where a real result set exists, so every early return leaves it false. It has to be the first thing done to the dict: a return added above that line would leave the key absent. ABSENT IS A THIRD STATE AND IT DEFAULTS TO TRUE. A caller that passes no report cannot know, and the safe reading there is the old behaviour. Only a real search can report False, so absent means "nobody asked" and never "it failed" — which is also why 66 existing mocked searches across twelve test files keep working unchanged rather than being rewritten to simulate a flag they do not care about. A FAILURE IS NOT MADE INVISIBLE. semantic_search_notes already logs a WARNING on a query failure, which is where a broken search belongs: a counter cannot say "I am broken" without a reader already trusting it. Three tests, and the middle one is what makes them discriminate — a blanket `return` passes the first and fails the second, because a call that searched and came back empty is the only evidence a threshold is too high (#3497). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 12:47:46 -04:00
bvandeusen	1c4ace5199	Merge pull request 'write_path stops reporting records it withheld itself as near misses' (#146) from dev into main	2026-09-09 00:44:44 -04:00
bvandeusenandClaude Opus 5	277f5df515	fix(telemetry): write_path reported records it withheld itself as near misses (#3739) ... Found by verifying the previous fix on live data — the check that fix was meant to make possible. write_path near_misses.max 0.822 p90 0.7521 top_score.min 0.6857 so the bar is at or below this A "rejection" that outscored every acceptance, and not one outlier: the p90 is above the bar too. #3739's fix keyed on `suppressed_count`, and I justified its NULL branch as "null means the caller passed its exclusions INTO the search, so the score is already post-exclusion". That holds for auto_inject and reuse_slot, both of which log the RAW search output and do their Python filtering after. It does not hold for write_path, the one note arm that filters TWICE: `exclude_ids` takes `seen - pulled_seen` into the search, but the pulled-and-seen ids stay in the query on purpose — the arm's query doubles as the resemblance test — and are dropped afterwards in Python. So the row carries a POST-filter count beside a PRE-filter score. The suppression column cannot rescue it the way it does for the rule arms. This arm's count would be PARTIAL — covering the drops made here and not the ones `exclude_ids` made inside the search — and a partial number under a name that reads as complete is the substitution this milestone exists to stop. So it reports null whenever its own filter removed anything: not measured on this call, because the bar was not the only thing that turned something away. Calls that withheld nothing keep reporting, which is most of them. Both directions are asserted. Without the second test, setting the field to null unconditionally would pass the first while deleting the measurement #3670 was built for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 00:34:11 -04:00
bvandeusen	6495236569	Merge pull request 'A repeat is not a rejection, and a compaction is not knowledge' (#145) from dev into main	2026-09-09 00:13:01 -04:00
bvandeusenandClaude Opus 5	ab14f783e1	chore(plugin): mint 2026.09.09.0408 — the hook change has to reach the cache (#3749) ... The manifest gate caught this, which is what it is for: FAIL plugin content changed but the version is still 2026.09.04.0140. An install has two halves and only one self-updates. The marketplace clone pulls on its own; the cache that actually EXECUTES refreshes only when this string changes. So a hook edit shipped without a bump reaches the repo and stops there — and the obvious debugging move, inspecting the clone, shows the fix present while the broken copy keeps running. That is #2209, #1040 and #2220, and the only detector was the operator saying "I don't think it updated". Minted with scripts/mint_plugin_version.py rather than hand-edited: the plugin ships straight from the repo with no build step, so there is no moment at which CI could stamp a value, and the script is the path the mint guard pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 00:08:55 -04:00
bvandeusenandClaude Opus 5	1cfbf43ccd	fix(plugin): the rule ledger clears when the context it describes is destroyed (#3749) ... The prior-art and tool-rule hooks record every rule id they have named in <state>/<sid>.rules.ids and hand it back as exclude_rule_ids, so a rule is surfaced once per session and then goes quiet. That is correct while the session still HOLDS what it was told. A compaction breaks it in the worst available way: it summarizes the earlier injections out of context and does not touch the filesystem. The rule ends up absent from context AND still excluded — unreachable for the rest of the session. The compaction banner this hook already prints tells the model to re-pull its ALWAYS-ON rules, but a rule an arm surfaced is conditional and is not in that set, so it has no other way back. The rules most likely to be in that state are the ones that fire most often. The stale ledger is genuinely found again rather than orphaned: the etag marker further down this same hook is rewritten on `compact` and keyed by session_id, which is only meaningful if the id survives a compaction. CLEARED ON THE SOURCES THAT DESTROY CONTEXT, AND ONLY THOSE. `compact` and `clear` destroy it while the file survives. `resume` does not — the context came back intact, so clearing there would re-surface every rule after a restore that lost nothing, which is the same defect from the other side. `startup` is a no-op against a new session id. `fork` keeps it, and the answer holds whichever way forks are keyed: a fork carries the conversation, so an inherited id means an accurate ledger and a new id means an empty file. Only the RULE ledger. The same directory holds .ids / .sync.ids / .derive.ids for the note arms; whether a surfaced note should return after a compaction is a different question with a different answer, and a `rm` glob would have decided it silently. The guard pins the DISCRIMINATION, not the deletion: the whole source table is asserted in one statement, so a blanket delete (all False) and a no-op (all True) both fail, and neither can be made to pass by editing one case. A second test pins the scope against that glob, and a third proves an event with no session id clears nothing rather than falling back to a wildcard. Runs with no SCRIBE_URL/SCRIBE_TOKEN on purpose — the clear is local, keyless and networkless, and must still happen against an unreachable instance. That is also why it sits above the config read rather than inside the dynamic tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 00:04:33 -04:00
bvandeusenandClaude Opus 5	a165483b92	fix(telemetry): a repeat is not a rejection, and near_misses counted it as one (#3739) ... Caught on the first live read after deploying #3670. The readout contradicted itself: pre_tool_rule top_score.min 0.7204 the lowest score ever RETURNED near_misses.max 0.7457 "rejected", but scored higher `best_available_score` is measured pre-threshold, which is right, but for the rule arms it is also PRE-EXCLUSION, which is not. The note arms pass `exclude_ids` into semantic_search_notes so their score is already post-exclusion and clean; `semantic_search_rules` takes no such parameter, so the rule arms filter in Python after the search and a rule that cleared the bar and was dropped as a repeat still reported its score on a zero-result row. That is #3497's distinction — a ranker decline versus a reader already ahead of it — reintroduced one level up, inside the field built to replace a tautology. The population now also requires `suppressed_count IS NULL OR = 0`. The NULL arm is principled rather than permissive: null means the caller filtered INSIDE the search, which is exactly the case where the reported score cannot be contaminated. Deliberately conservative — a call carrying both a repeat and a lower genuine miss is dropped whole, losing that point. It undercounts; it cannot corrupt, which is the right way round for a number read against a bar. It also makes `near_misses.max < threshold` true BY CONSTRUCTION rather than by fixture: an above-bar candidate nobody excluded would have been returned, so its call is not in the population at all. THE TEST DID NOT CATCH THIS, and that is the part worth keeping. The assertion `nm["max"] < 0.72` was already there, with exactly the right intent. It passed because the fixture contained no suppressed call — the guard held because the breaking shape was absent, not because the code was right. Rule 167's stated failure mode, in a test written while citing rule 167. The fixture now builds that shape: a 0.9 hit dropped as a repeat, which lands in the population and drags `max` above the threshold unless the predicate excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 16:42:51 -04:00
bvandeusen	53df6742cb	Merge pull request 'Milestone 379 closes: the readout can now say what it did not measure' (#144) from dev into main	2026-09-08 16:06:14 -04:00
bvandeusenandClaude Opus 5	e7c1af32a0	fix(telemetry): the bar can only be judged from what it rejected (#3670) ... `cleared_threshold` was documented as the number to read first. It was a tautology. The search applies the threshold before returning, so every returned result cleared it by construction and a call with no results has no top_score to compare — the condition was true exactly when `result_count > 0`. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen source/window readings ever taken, today's live seven included. The reading procedure built on it asked the reader to compare a number with itself, and a threshold change was unobservable through it: raise the bar and both numbers move together, so the field could never show a bar set too high. REPLACED, NOT JUST REMOVED. The question the table exists to answer is whether the bar is in the right place, and that is only answerable from the calls that returned NOTHING: how close did the best rejected candidate come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair; the same bar turning away 0.30s is working. Both render as a zero-result call today and nothing separates them, because the losing score is discarded inside the search. So both searches now rank WITHOUT the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one, and an over-fetch that returned N above-bar rows returns the same N plus some losers. What changes is that the losers are visible instead of dropped in the query. `report` carries the score out without changing what a search RETURNS: eight of eleven call sites want hits and nothing else. New column (migration 0096), nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence, which is the substitution this whole milestone corrects. The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step from the shape that produced #2663, where a rejected query was swallowed by the broad except and every counter read zero. It carries an integration guard for that reason: only real Postgres can say it parses, and the symptom of failure is silence. Also adds a guard that no int field in a bucket equals `calls - zero_result_calls`. That identity is what `cleared_threshold` satisfied for its whole life, and it survived because it had its own name and nobody added the two numbers beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:50:05 -04:00
bvandeusenandClaude Opus 5	277aea58e4	test(telemetry): pin the identity that falsified this milestone (#3668) ... Reverts the deliberate break from 5e19a1b. `plugin_context.py` is now byte-identical to before it; the only change against that baseline is the guard itself. FALSIFIED, not argued (rule 167). CI run 6046 with `results=hits` in both arms failed exactly the predicted four cases — [one-already-held-write_path] FAILED [one-already-held-pre_tool] FAILED [all-already-held-write_path] FAILED [all-already-held-pre_tool] FAILED [nothing-held-*] passed — and the nothing-held cases passing is the point, not a gap: with no exclusions both recorders see the same list however wrongly they are wired, so that case can never discriminate and a guard built only from it would read as coverage while catching nothing. WHAT IS PINNED. Both arms build one `fresh` list and hand it to two recorders in one function, so the call log and the surfacing log cannot disagree about what a single call showed. Ids, not counts: equal counts drawn from different lists is a real way for this to break, and a count comparison would call it agreement. Three hits, where production returns at most one. The identity holds at any limit because both recorders read the same list, and stating it that way survives RULEHINT_LIMIT moving again — it has moved once already (2 → 1, 2385100), and that move is half of why the original reconstruction misread its own numbers. NOT DONE, deliberately: the readout-level self-check the task also proposed. `cleared_threshold` counts CALLS that beat the bar while `surfaced` counts RULES, so that identity holds only while the limit is 1 — it would fire on a healthy system the moment the limit rises. The arm-level form has no such coupling, which is why the guard lives here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:36:03 -04:00
bvandeusenandClaude Opus 5	5e19a1b028	test(telemetry): FALSIFICATION — prove the identity guard can fail (#3668) ... Deliberately broken, reverted in the next commit. Rule 167 requires every guard be falsified against the regression it names before it is trusted, and rule 10 puts CI as the only place that can run it — so the failure has to be made to happen here rather than argued for. The guard: both rule arms feed one `fresh` list to two recorders, so the call log and the surfacing log cannot disagree about what one call showed. The regression: `results=fresh` becomes `results=hits` in both arms, so the call log counts what the ranker found while the surfacing log counts what was shown. That is not a hypothetical shape. It is exactly the divergence that would make a correct system report a lost write when the two tables are later compared in aggregate — the reading that scoped this milestone at five steps against a defect that did not exist. Expected red: the one-already-held and all-already-held cases on both arms. The nothing-held case must still PASS — with no exclusions both recorders see the same list however wrongly they are wired, which is why it could never have been the discriminating case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:31:49 -04:00
bvandeusen	be08edcd7e	Merge pull request 'Telemetry coverage, and the rule arms stop filtering to one tier' (#143) from dev into main	2026-09-08 11:11:40 -04:00
bvandeusenandClaude Opus 5	7a2aff7bc1	fix(telemetry): a surface that stopped recording is not one that never ran (#3720) ... `out["sources"]` was built only from the windowed aggregate, so a source with rows in `retrieval_logs` but none inside the window got no bucket at all. Absent is exactly how a source that never existed renders, so a surface that WAS recording and went silent became unreadable — #2663 one level up, the failure that looks like the correct answer. Two queries at different scopes, and only one shaped the output. `_complete_from` reads all-time and knows every source the table has ever held; the windowed loop dropped whatever it did not return. Every such source now gets a zero bucket. Zero is a real measurement here rather than a manufactured one: the all-time query proves the source was recording, and it made no calls across a window it fully covers. No `covers_window` special case is needed either — a source whose first row fell after `since` would have that row IN the window and already hold a bucket, so anything reaching this branch began before it. The counts are 0 and everything else is null. A sampled distribution is not the same claim as a call count, and rendering p50 as 0.0 for a source nobody sampled would assert a measurement — #3311's mistake, in the readout built to prevent it. Found while fixing #3712's fixture, which failed with KeyError for this exact reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 11:07:21 -04:00
bvandeusenandClaude Opus 5	0808e8259a	test(telemetry): the old surface needs a row in the window to have a bucket at all (#3712) ... The per-source grain test gave `auto_inject` a single row 90 days back and `pre_tool_rule` one 2 days back, then asserted on both buckets. Only the young arm got a bucket: `out["sources"]` is built from the WINDOWED query, so a source with no rows inside the window is absent entirely, and the assertion died on KeyError before it could test anything. `complete_from` and the bucket come from different queries — all-time for the first, windowed for the second — and the fixture only satisfied one of them. Gave `auto_inject` a second row inside the window, which is also the shape being described: an old surface that is STILL recording. The 90-day row still sets its `complete_from`. Still discriminating: auto_inject reads True and pre_tool_rule False, and a per-table `_complete_from` would make both 90 days and fail the second assertion — the regression this test is for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 10:39:42 -04:00
bvandeusenandClaude Opus 5	950c93c5d4	fix(telemetry): the coverage helpers were defined inside the function they serve (#3712) ... `_complete_from` and `_coverage` landed between `retrieval_summary`'s docstring and its body. Python does not object to that the way it looks like it should: blank lines do not close a block, so the whole remaining body — indented four spaces, sitting after `_coverage`'s `return` — became unreachable code INSIDE `_coverage`, and `retrieval_summary` became a function that is nothing but a docstring. The error surfaced three ways at once, none of which named the cause: fourteen F821s for `days` and `user_id` (real: those are `retrieval_summary`'s parameters, and the body no longer lived there), a SyntaxError on `async with` (real: `_coverage` is sync), and every test module that imports this file failing to collect. Moved both helpers above `retrieval_summary`, beside `_bucket` and `_round`, where the file's other helpers already are. No behaviour change — this is the code that was meant to be there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 10:34:57 -04:00
bvandeusenandClaude Opus 5	21a5831479	feat(telemetry): every counter says when it started being recorded (#3712) ... A window that opens before a counter existed reports that counter as though it had been measured throughout. The reader cannot tell "zero because nothing happened" from "zero because nobody was counting yet", and — worse — cannot tell a partial count from a complete one. That middle case yields a plausible FRACTION rather than an obvious zero, which is what makes it dangerous. It is not hypothetical. A 7-day window opened while the ranked rule surfacing recorders were four days old produced an apparent 64% write loss, which survived a code review, four ruled-out alternative causes and a five-step milestone before an identity check falsified it in one read. Every counter block now carries `complete_from` and `covers_window`. THE GRAIN IS THE SOURCE. retrieval_logs accumulates for months, so a per-table earliest row says months for every source it holds — including an arm added days ago whose counter means something else entirely. The old source would vouch for the young one, which is the exact reading this prevents. A SECTION TAKES ITS LATEST CONTRIBUTOR, NOT ITS EARLIEST. A figure summing several sources is complete only once every one of them was being written, so "*" is a max. Using min would reproduce the original error in miniature. `covers_window` is null, never false, when nothing was ever recorded: "no measurement" is not "partial measurement" — the null convention #3497 established for `suppression`, one level up. Also corrects a stale claim in the tool docstring: it still taught readers that write_path_rule "has never once declined to fire" (#3311). That was the arm writing its retrieval_logs row only on calls that found something; #3497 fixed it, and the arm declines the large majority of its calls. _complete_from takes the caller's session rather than opening its own, departing from the services canon (#2860) because it runs inside an existing block; to be recorded against the ledger once it ingests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 10:29:02 -04:00
bvandeusen	4a85220aea	Merge pull request 'feat(rules): the rule arms stop filtering the corpus to one tier (#3702)' (#142) from dev into main	2026-09-08 00:31:12 -04:00
bvandeusenandClaude Opus 5	dd1e6e2645	feat(rules): the rule arms stop filtering the corpus to one tier (#3702) ... Both arms passed tier="conditional", on the reasoning that an always-on rule is already in the session so re-surfacing it is pure noise. That conflates two different things: PRESENT IN CONTEXT the rule was delivered at session start SALIENT AT THE MOMENT the rule is in front of the reader when the action it governs is about to be taken A rule handed over in a list at turn zero is present while a session writes a config value three hundred turns later. It is not surfaced. So the filter did not skip a redundant hint — it made a whole class of rules permanently ineligible for the only mechanism that puts a rule in front of an agent AT the moment, and the more important a rule is, the likelier it sat in that class. Underneath, the filter was doing the THRESHOLD's job. Whether a rule belongs in a hint is a relevance question and a similarity bar is the control for relevance. A categorical exclusion standing in for a relevance judgment cannot be tuned, cannot be measured, and cannot be wrong in a way anybody notices. MEASURED, NOT SETTLED. The old comment's fear is real: a hint that fires on every write and says obvious things teaches the reader to skip the block. It had simply never been checked, and retrieval_logs already records the scores to check it with. Rules clearing often and high means the fear was justified and the BAR is the work; rules clearing rarely in a thin band means relevance was always sufficient. Only eligibility moved. The bar stays at 0.72 and k stays at 1, so the resulting distribution has one cause — and k=1 bounds the blast radius: a wider pool can change which rule surfaces and how often, never how long a single hint gets. The threshold rationale's first premise ("the eligible corpus is TINY — tier=conditional only") is updated rather than deleted: a larger pool makes clearing the bar mean MORE, so that argument weakened, and the bar was left alone anyway rather than move two variables at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 00:24:02 -04:00
bvandeusen	14ff41faf5	Merge pull request 'feat(rules): rule creation becomes propose-then-approve (#3557)' (#141) from dev into main	2026-09-04 22:01:47 -04:00
bvandeusenandClaude Opus 5	c3ecdf0972	feat(rules): the rule gate becomes a practice with a question, not a prohibition (#3557) ... The first cut opened "NOT YOURS TO CALL UNPROMPTED", and that is the wrong instrument. A caller reading a prohibition stops NOTICING rule-shaped things rather than noticing them and asking — which trades a small failure for a larger one. The wanted behaviour is more proposals, not fewer. So both docstrings now describe the practice: propose readily, state the four things, and close with a question the operator answers in one word — approve it as written / let's talk about it / no. Named options where the interface has them, three written-out options where it does not. "Approve it as written" is what makes element 1 load-bearing: they approved TEXT, so that text is stored verbatim. "Let's talk about it" is framed as the expected answer rather than a setback. "No" routes the observation to create_note, which records without binding. The argument for asking is also better than consent. The operator's yes is the one moment the rule is certainly in front of them: afterwards a conditional rule is not read aloud at session start, and a project rule is absent from an unfiltered list_rules(). The proposal IS the review. Guard gains the answers-offered-back element and drops the wording that forbade; its header records why the framing changed, so the prohibition does not get reintroduced as a tidy-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-04 20:11:42 -04:00
bvandeusenandClaude Opus 5	1a34363059	feat(rules): the rule-creation tools ask for approval, and ask what would enforce it (#3557) ... Every gate on create_rule and create_project_rule was about SHAPE — rule vs process vs snippet, one-thing-you-could-violate, general-enough, not-a-dupe. All of them improve a rule someone has already decided to write. None asked the prior question: has the person this will bind agreed to be bound by it? Both docstrings now open with the gate, and with the four things a proposal carries: what it would require in the words it would carry, its intent, why now, and how it would be enforced. The fourth is the one that decides it. "A test, a CI check, a hook, a schema constraint... or nothing" is a question that sometimes dissolves the rule: what a test can assert should BE that test, and a rule is what is left when nothing mechanical can hold the thing. A rulebook grows by default and shrinks only on purpose. create_project_rule needs the gate more, not less, and says so: a project rule is absent from an unfiltered list_rules(), and a conditional one is absent from session start too, so one written there can bind for months without ever having been in front of the person it binds. test_rule_creation_asks_first pins structure, never wording — each element matches a family of synonyms, and the gate must precede the Args: block, because a caller who has decided to make the call reads the parameters and not the prose under them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-04 20:02:05 -04:00
bvandeusen	67df41ae00	Merge pull request 'feat(rules): the instruction surfaces say to RETRIEVE a rule, not only to receive one (#3523)' (#140) from dev into main	2026-09-03 22:07:10 -04:00
bvandeusenandClaude Opus 5	30d87e461a	feat(rules): the instruction surfaces say to RETRIEVE a rule, not only to receive one (#3523) ... Every instruction surface told a session to load the always-on rules and stopped there. None said the loaded set is partial, so an empty one read as "no rule applies" when it only ever meant "none was pushed" — different claims, and only one of them has been checked. That is #2198's asymmetry one level in. The earlier defect was trusting the SessionStart push over the explicit pull; this is trusting the resident TIER as if it were the whole rulebook. It is also why the always-on tier was the only one that worked, on any install rather than this one (rule 115): a rule nothing retrieves must be resident to bind at all, and a resident rule costs tokens in every session forever — so a rulebook that only delivers cannot grow past what one session holds. Retrieval lifts that ceiling, and it fires only if something asks. A tool-choice reflex asks least of all (#3476, #161). The same obligation now lands on all three session-start surfaces, because rule 119 makes them the specification jointly and a surface stating it differently IS the product behaving differently (#2497). Pinned by test_every_session_start_surface_states_the_conditional_retrieval, mirroring the pull test beside it. THE BUDGET TRADE. _INSTRUCTIONS sat at 1978 against a 2000 test budget, and its own comment says an addition there is a trade, never an append. Bought the new clause by trading out "Processes are saved procedures (follow verbatim)" and "Deletes are trash-recoverable" — both already in DISPLACED_TOPICS and already stated on a delivered surface, and both per-tool guidance, which by this block's doctrine belongs in the tool docstring. Now 1976. Recorded in the comment above the block so it is not silently reversed. Plugin version minted: shipped plugin content moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 21:41:53 -04:00
bvandeusen	0915c48bb0	Merge pull request 'feat(telemetry): tell a ranker decline from a repeat before the observation window opens (#3497)' (#139) from dev into main	2026-09-03 21:23:01 -04:00
bvandeusenandClaude Opus 5	8be555d6dd	feat(telemetry): tell a ranker decline from a repeat before the observation window opens (#3497) ... Making the rule arms log every call exposed a second ambiguity in the same row. `result_count == 0` is two unrelated events wearing one number: - the ranker found nothing above the bar — the only evidence a threshold is set too high; and - the ranker found only what this session had already been shown — which says nothing whatever about the bar. A long session excludes its way into the second, so the arm reads worse the longer it runs correctly. Rows written now carry the ambiguity permanently, which is why this lands before any watch period rather than after. `retrieval_logs.suppressed_count` (0095, nullable) holds what the caller dropped as already-shown. Both rule arms report it; they filter in Python and always know. The note arms pass exclusions INTO semantic_search_notes and never see what was dropped, so they store NULL. THE NULL IS LOAD-BEARING. It means "not measured here", and the readout renders it as `suppression: null` rather than a zeroed dict. Defaulting to 0 would let an unmeasured surface read as a perfectly clean one — the same substitution of an artifact for a measurement that #3311 made. No backfill, for the same reason: existing rows genuinely do not know. `retrieval_telemetry`'s `sources` gains `suppression` with `measured_calls`, `calls_with_suppression` and `zero_because_already_shown`; subtract the last from `zero_result_calls` for the true ranker declines. The MCP tool docstring says to read the two together and warns against reading the null as a zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 21:12:38 -04:00
bvandeusen	aea7b63b62	Merge pull request 'fix(telemetry): both rule arms logged only their hits, so the clear-rate could only read 100% (#3497)' (#138) from dev into main	2026-09-03 07:19:08 -04:00
bvandeusenandClaude Opus 5	48804c437d	fix(tests): the write-path telemetry test asserted the defect, not the split (#3497) ... `assert_called_once` held only because the rule arm skipped its retrieval_logs row when it found nothing. With the arm logging every call, the test now asserts what it was always about — exactly one `write_path` row, no `auto_inject`, and the rule arm keeping its own separate source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 06:59:52 -04:00
bvandeusenandClaude Opus 5	154a5de13e	fix(telemetry): both rule arms logged only their hits, so the clear-rate could only read 100% (#3497) ... `write_path_rule` reported `zero_result_calls: 0` and `cleared_threshold: 133/133` — a perfect record no other surface comes near (`write_path` 421 zeroes of 613, `reuse_slot` 124/199, `auto_inject` 114/326). #3311 read that as a measurement and milestone 333 was scoped on it. It was an artifact. Both arms called `record_retrieval` inside a guard on having results — the write-path arm behind `if fresh:`, the pre-tool arm below `if not fresh: return out` — so a call that found nothing wrote no row. The statistic was a fact about the shape of the code, true at any threshold whatsoever. The call log moves out of the guard in both arms. The surfacing log stays in it: nothing was shown, so no surfacing occurred. `results=fresh` is kept deliberately — the note arms pass exclusions into `semantic_search_notes`, so what they log is already post-exclusion, and logging `hits` here would make this row mean something other than every other row in the same readout. The defect bites hardest on the pre-tool arm, which fires on every Bash call: with no rows at all, a ranker that declined is indistinguishable from a hook that never fired — the silent failure the arm exists to stop. Tests cover both arms behaviourally (found nothing; found only what the session already held; searched nothing at all, which must stay silent) plus a structural guard, because this was one level of indentation and it appeared independently in two places. #3311 and the `rule_usage` docstring corrected rather than quietly rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-03 06:57:13 -04:00
bvandeusen	5b02908dfd	dev → main: rules become measurable at the preload, and retrievable at the tool call (#137)	2026-09-02 23:41:09 -04:00
bvandeusenandClaude Opus 5	2ee24b9d2b	feat(rules): rules before tools — a PreToolUse arm keyed on the action (#3476) ... The only just-in-time rule surface was registered on `Write|Edit` and queried with `code or path`, so a rule could be retrieved at the moment of a code write and nowhere else. Every rule about which tool to reach for — don't curl the forge, don't stand up a stack, don't run the suite locally, don't branch — was unreachable exactly when it mattered, and residency in the always-on preload was the only surface it had. That is the pressure that grew the resident set to 31 against #3089's ceiling of ~23; it was never a judgment anybody made. A reflex generates no query, so an instruction to check the rules cannot catch one. A mechanical trigger can: the tool call IS the query, and a reflex has to become a tool call before it can do anything. `build_tool_rule_hint` is deliberately tool-agnostic — a name and a string — so widening the matcher later is a hooks.json edit with no server change. The hook starts on Bash, which is where the action reflexes live. The two pre-tool arms share ONE session ledger of already-named rules (`<state>/<sid>.rules.ids`). Two ledgers would mean a rule named by one arm gets re-offered by the other, and the hint that fires most often is exactly the one that must not repeat itself. A test asserts both scripts build the same path, and another checks the shell hook and the Python route agree on every query-arg name (rule 33) — a rename there fails silently, looking like a surface that never finds anything rather than a broken one. Deliberately silent on outage, unlike the prior-art hook: a write is occasional, a Bash call is not, and an outage line before every command is what gets a channel muted. `tier="conditional"` matches the write arm and is the transition point — an always-on rule is already resident, so re-tier one and it starts arriving here instead of in every session's preamble. `pre_tool_rule` joins RANKED_SOURCES: this arm chose what it showed, so a pull can settle whether the choice landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-02 23:31:22 -04:00
bvandeusenandClaude Opus 5	8b9b3a1d9b	feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473) ... The ranked rule arm became measurable in M333. The preload did not — and that is the surface whose value is actually in question. `list_always_on_rules`, the SessionStart block and every `rules_payload` caller handed rules over wholesale and emitted nothing, so the resident set's token cost was certain and its usefulness could not be tested even in principle. Bulk deliveries now record as AMBIENT, beside the ranked count and never inside pull-through. Folding them in would mean growing the always-on set depressed the arm's measured precision and trimming it flattered the arm, neither for any reason to do with the arm. `RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there is one ranked rule source and this change adds seven bulk ones, so naming the rare half makes a forgotten surface default to ambient — under-counting it — rather than padding the denominator with surfacings nobody chose. Two lookalike call sites are deliberately left silent, with a test to keep them that way: the write-path etag arm and `rules_etag_for` read the rules to build or compare a MARKER and show nobody anything. No migration — `event` and `source` are plain Text with no CHECK (rule 36 does not apply). Snippet #2858 updated to the new `rules_payload` contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-02 23:06:40 -04:00
bvandeusen	34cd389371	dev → main: rule usage telemetry, the plugin's derived version (#136)	2026-09-02 18:52:20 -04:00
bvandeusenandClaude Opus 5	6627cfc2f0	feat(rules): a usage badge on the rule list, and the badge becomes canon instead of a second copy (#3319) ... Milestone 333 step 5, and rule 27 — the counter had a tuning point from step 4 and no operator-facing one until now. The task said to reuse the snippet badge's classes rather than mint a parallel set, citing the eight duplicated CSS families the ledger already carries (#3207). `.usage-tag` lived in SnippetListView's SCOPED block, so "reuse" was not available: copying it into the rule pane would have been the ninth family, and importing it is not a thing a scoped block permits. So it was promoted rather than copied. Three pieces, each of which existed once and now exists once: - `components.css` gains `.usage-tag` / `.usage-dead`, geometry and colour only, with the scoped original deleted rather than left behind. - `UsageBadge.vue` holds the logic the two lists would otherwise duplicate — the >=3 dead-weight threshold, the empty-string-renders-nothing rule, the tooltip. - `types/usage.ts` holds `RecordUsage`, one client type over two tables. `SnippetUsage` becomes an alias, so no existing consumer changes. THE ADVICE IS A PROP, and that is the substance rather than the plumbing. The counts read identically for every kind; the remedy does not. A snippet offered and never opened should probably be rewritten or deleted — one action. A rule in the same position has TWO possible causes and the operator has to pick: its trigger may fire on the wrong work, in which case `when_to_apply` wants rewording, or it may genuinely not be wanted. Baking "delete it" into the component would give the wrong nudge half the time on the surface where being wrong is most expensive, since a deleted rule stops binding behaviour. The route zero-fills every row through `usage_for_rules`, one aggregate per page — per-row would be N+1 by construction. That matters more here than for snippets: every rule on every existing install predates `rule_usage_events`, so the zero-filled shape IS the common case for a while, and a route that attached the key only where it found events would leave the badge reading undefined on almost every row. `usage_for_rules` had no test at all — step 1 covered the write path and the zero shape and left the aggregate uncovered, which only became load-bearing when a list started rendering it. It now has an integration test over real Postgres, including that a rule with no events comes back zero-filled rather than absent. Recorded as snippet #3460, per the design system's own instruction that the component layer lives as snippets rather than as prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 18:37:55 -04:00
bvandeusenandClaude Opus 5	238510080e	feat(retrieval): the standing-rule arm gets its own bar, and asks for one rule not two (#3318) ... Milestone 333 step 4 — the split #2223 made one surface down, now made for the third corpus. The arm inherited WRITEPATH_DEFAULT_THRESHOLD = 0.68, a number measured against code-vs-note-PROSE and never re-derived for code-vs-RULE-TEXT. THE DEFAULT IS ARGUED STRUCTURALLY, NOT READ OFF A HISTOGRAM (rule 115). Two facts hold on any install, including one with six rules and no telemetry: - The eligible corpus is tiny — conditional rules only, a handful to a few dozen against thousands of notes. A top-k over forty candidates always returns something, so "the best match cleared the bar" stops meaning "a good match exists". A bar calibrated for best-of-thousands is cleared by best-of-forty as arithmetic, not relevance. - Rules are short imperative technical English, far more homogeneous than note prose. #2223 put the code-vs-prose floor at 0.55-0.63 and set 0.68 above it; a more homogeneous corpus has a HIGHER floor, so 0.68 is not merely inherited, it sits below where this corpus's noise lives. 0.72 errs deliberately toward silence on an asymmetry that is also structural: this hint fires on EVERY write. A missed rule is recoverable — it is still in Scribe and the agent can search it. A hint that cries wolf is not: it teaches the reader to skip the whole block, and the true positives go with it. The arm's own comment already said "noise on a hint that fires on every write is how a hint gets ignored". Pinned as an INEQUALITY, not a value: test_the_rule_bar_defaults_above_the_code_bar asserts RULEHINT > WRITEPATH, so tuning the number stays free while inverting the relationship — which would silently reinstate #3311 — does not. RULEHINT_LIMIT = 1, and deliberately not a knob. With a corpus this small, k=2 means the second line is almost always the second-best noise wearing the same confident framing as the first; halving k halves that regardless of the bar. It stays a constant because it is a decision about how loud one hint may be, not a per-install tuning question — and a knob nobody turns only adds a way to misconfigure the surface. Reachable from Settings, no restart (rule 25), with copy that says which way to move it and points at retrieval_telemetry's rule pull-through — which step 3 made readable — to tell "arriving unread" from "never arrived". Every config stand-in in the suite gained the key, not just the one that noticed. The arm reads `rule_threshold` while BUILDING its search arguments, so a missing key raises inside its fail-open except and turns the arm into a silent no-op — indistinguishable from it running and finding nothing. That is the same vacuous-pass shape that bit step 2, one layer down (rule 33). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 18:05:00 -04:00
bvandeusenandClaude Opus 5	8901c904a9	feat(telemetry): retrieval_telemetry reports rule pull-through where it reported nothing (#3317) ... Milestone 333 step 3, the read half. Steps 1 and 2 built the table and filled it; until now nothing read it, and `usage` — sourced entirely from note_usage_events — described notes only while `sources` happily listed a write_path_rule row above it. A reader takes the aggregate as covering everything named above it. It did not. A SEPARATE `rule_usage` BLOCK, not folded into `usage`. Two reasons, and the second is the one that bites: the corpora differ by orders of magnitude, so a blended ratio would be the note ratio with noise on it and the rule arm would stay invisible inside it; and `usage` is what existing callers already read and compare across windows, so silently changing what it counts would move a number nobody was told had changed meaning. There is a test asserting rule events stay out of the note block. No `ambient` key, unlike the twin. Nothing surfaces a rule un-ranked — list_always_on_rules and enter_project hand rules over wholesale but emit no event — so there is no ambient class to subtract. The absence is a fact about the data, not an oversight, and it returns when a bulk loader starts emitting. Guarded separately, like `by_source`. This table did not exist a commit ago, and an instance running upgraded code against un-migrated schema would otherwise take down two readouts that work perfectly in order to report a third that cannot. On failure the FLAG is added and the SHAPE is kept — a caller must not have to choose between crashing on a missing key and quietly rendering zeros it has no right to. `pull_through` is None rather than 0.0 on an empty window, matching the note block. A ratio of zero asserts "rules were shown and none opened"; with an empty numerator and denominator that is a claim the data does not support, and it is the reading that would make a brand-new install look like a broken one. Also fixed, from #3311: the rule arm never timed its search, so it was the one source in the readout reporting a null p90_duration_ms — a gap that reads as "this surface is somehow not measurable" rather than "nobody passed the number". Both docstrings updated in the same change. The tool's is the agent-facing contract (rule 119) and it explicitly said rule surfacings were absent and had "no usage counter at all". Leaving that would have had a reader conclude the arm has zero pull-through rather than a separate one. Tests are integration for the reason the block above them is: real GROUP BYs and count(distinct) against a table a commit old, in a module whose one production outage was a SQL shape the database rejected inside a broad except. A mock would agree with whatever the code does, including nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN	2026-09-02 17:25:11 -04:00