Retire the always-on tier — every rule arrives by retrieval (milestone 394) #152

Merged

bvandeusen merged 13 commits from dev into main 2026-09-11 16:43:20 -04:00

Conversation 0 Commits 13 Files Changed 64

+1889 -1822

13 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	32db56c0df	fix(394): the co_surfaces test could no longer fail, so it was repaired not relaxed ... Last run left one failure, and it was the useful kind: the partner ARRIVED (the scoping fix in 9c5ab1d worked) but carried no `via`, because it came through the ordinary query rather than being dragged in by the edge. The assertion was about to be read as "the edge is broken". It was not. The edge was never exercised: since 394 an untagged rule in a subscribed rulebook applies on its own, so the partner was already applicable and there was nothing left for `co_surfaces` to do. The test had quietly stopped testing anything — passing the first assertion for a reason unrelated to the mechanism it names. Rule 167's case exactly, so the fix is to restore its ability to fail rather than to soften the assertion. The partner is now TAGGED to an area this project does not work in, which puts it out of reach of everything except the edge, and the test asserts that unreachability before drawing the relation. If the edge ever stops dragging partners in, this fails again — which it could not have done a commit ago. The neighbouring suppression test keeps an UNTAGGED partner, deliberately and now explicitly: that one is about a suppression outranking an edge, so its partner should be reachable by every route, not none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 16:38:14 -04:00
bvandeusenandClaude Opus 5	9c5ab1d6ad	fix(394): subscription is the scope — areas narrow only where an author asked ... I got this wrong in 0e10f6b and the integration suite caught it. get_applicable_rules filtered `always_on OR area-reachable`. Removing the tier, I kept only the reachable arm and argued that a project with no canonical-tagged Systems should get no bulk rules and reach them by retrieval instead. Two things wrong with that. The query is ALREADY scoped to rulebooks the project SUBSCRIBED to, so the project had opted in and was then handed a subset of what it asked for — subscription is not bulk delivery, it is the opt-in. And milestone 394 is explicit that subscription-derived rules are not the always-on tier and are not what it removes; I narrowed something the milestone said to leave alone. The failure that surfaced it is a good one: a co_surfaces partner never arrived, because the rule it travels with had been filtered out before the edge could drag it in. A behaviour two steps from the change. What ships instead is narrower than "drop the clause" and wider than what I had: every rule in a subscribed rulebook applies, EXCEPT that a rule tagged to specific areas applies only to a project working in one of them. An untagged rule was never narrowed by anyone, so it is general to its rulebook by construction; a tagged one is an author saying "this is about CI" and meaning it. D7's deterministic narrowing is kept where it was asked for and not invented where it was not. Also: three tests covering the SessionStart preload, the always-on tool and the two marker paths — all surfaces that no longer exist — and the backup's declared-section list, which still named the section its table took with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 16:33:16 -04:00
bvandeusenandClaude Opus 5	8820551058	fix(394): unbreak collection, the TS typecheck and the plugin version ... Five validators failed on 0e10f6b; these are the ones the logs named. COLLECTION died first and hid everything else: test_inception.py still imported project_rulebook_exclusions, so pytest aborted before running a single test in either the unit or the integration lane. The migrations therefore never ran, which means 0099 and 0100 are still unverified — this push is what puts them in front of real Postgres. The obsolete table test went with the import, and its module docstring now says why rather than just describing one fewer thing. TypeScript: RulebookDetailPane's currentRulebook computed existed only to feed the always-on toggle, so removing the toggle left it unread; the vue 'computed' import went with it. Plugin version minted — plugin content changed and the manifest gates the executing cache (#2209), so the hook check fails until it moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 16:26:56 -04:00
bvandeusenandClaude Opus 5	4eebe271ed	fix(rules): restore system_ids and clear the imports the deletions orphaned (#394) ... Three defects from the sweep, all caught by ruff. system_ids was REMOVED FROM create_rule AND create_project_rule — a real API regression, not a lint nit. The parameter shared a signature line with tier, so deleting the tier deleted it too, and the tools lost the ability to tag a new rule to an area. Areas are what let a rule reach a project after this milestone, so the one parameter that decides reach went missing from the two tools that create reachable rules. The other two are imports left holding nothing: services/rulebooks.py's module-level datetime and its IntegrityError were used only by functions this milestone deleted, and plugin_context lost four (select, async_session, RulebookTopic, rulebooks_svc) with the preload and _topic_titles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 16:24:38 -04:00
bvandeusenandClaude Opus 5	0e10f6bb8a	feat(rules)!: retire the always-on tier — every rule arrives by retrieval (#394) ... Milestone 394, steps 5-8. Operator: "remove the always on rule functionality as the goal was to not have it at all since it didn't seem to work as expected." Unconditional preload had three failures the retrieval arms do not. It could not be MEASURED — a resident rule is in the context whether or not it mattered, so nothing distinguished "this governed the act" from "this was scenery", and it was the one surface structurally exempt from the scoreboard judging every other. It was SUMMARISED AWAY by compaction while the session went on believing it held the rules. And it CROWDED OUT the few rules that applied with the thirty that did not. WHAT GOES Schema (0100): rules.tier + ck_rules_tier, rule_versions.tier, rulebooks.always_on, and project_rulebook_exclusions — a table recording a project's opt-out of something that no longer binds it unasked. Tools: list_always_on_rules, exclude_always_on_rulebook, include_always_on_rulebook. Service: the same three plus rules_etag_for, _valid_tier and the whole etag family. The SessionStart preload and the write-path staleness arm go with them: nothing is resident, so nothing can have drifted since a session loaded it. THREE CALLS WORTH REVIEWING enter_project got NARROWER, not wider. Its filter was `always_on OR area-tagged`; dropping the tier arm leaves the deterministic half, so a project with no canonical-tagged Systems gets no bulk rules and reaches them by retrieval instead. Dropping the whole clause would have made that payload bigger than the preload this milestone deletes. Backups import tolerantly. A pre-394 archive carries tier, always_on and the retired inception choice; none is read, and the exclusion key is DROPPED rather than remapped, because restoring it would write data that validate_inception now rejects as unknown. The migration is irreversible in the way that matters and says so: downgrade recreates the columns at their defaults and cannot restore which rules were always-on. A value invented to fill a hole is not a measurement. THE INSTRUCTION SURFACES SAY THE HARDER THING Deleting "call list_always_on_rules()" is easy; replacing it is not, because the new model asks a session to trust something it cannot see. All three surfaces now say a session holds nothing, that rules arrive when work matches them, and — the half that got dangerous — that "no rule arrived" means "nothing matched", never "there is no rule". Under residency an empty session was rare and suspicious; it is now the ordinary state of most turns, so reading it as permission is wrong on nearly every turn rather than occasionally. That is #3720's defect at session scale. test_instruction_surfaces_agree is repointed rather than retired: its two halves collapsed into one instruction, and it gains a guard that every surface states what absence means. _INSTRUCTIONS is back at 1999/2000 — the inception clause paid for the longer HOW line. UI (rule 27, and the opportunity step 8 named) The tier selector is gone, and what replaces it is the point: `when_to_apply` is now the field that decides whether a rule is ever seen, so the editor marks it required, warns while it is empty, and both rule lists badge a trigger-less rule "never surfaces". A rule without one is not quiet, it is unreachable. TESTS Two files deleted outright — test_rules_etag.py and test_inception_rules.py tested subsystems that no longer exist. Elsewhere obsolete cases were removed and the rest repointed. One deserves naming: the wiring test asserted the act arms pass no `tier`, which had become an assertion that could not fail. It is repointed onto `kind`, which does still exist and where the same claim is live — a preference must reach a write exactly as a rule does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 16:22:17 -04:00
bvandeusen	18e3cf9f2f	fix(telemetry): scrub existing rows, and stop --author= being eaten (#3925)	2026-09-11 15:43:31 -04:00
bvandeusenandClaude Opus 5	bfa2d419f9	fix(telemetry): a logged query never carries a credential (#3925) ... pre_tool_rule retrieves against the RAW COMMAND TEXT and write_path_rule against the code being written, so whatever was on the command line or in the buffer is what record_retrieval stored in retrieval_logs.query. A command that exported a token stored the token. Storing it was not the worst of it. near_miss_samples is the readout the threshold docs tell you to open before moving a bar, so the value came back OUT into an agent's context on the next tuning pass — which is exactly how this was found, mid-way through #3853's threshold spike. SCRUBBED ON WRITE, at _build_payload — the single seam every source reaches the column through. A read-side filter would leave the secret in the table where a backup or a debug query still reaches it, and a per-caller scrub would be three places for one to be forgotten by whoever adds the fourth arm. REDACTED VISIBLY. `[redacted:<kind>]` rather than a silent deletion: a reader who cannot tell a scrubbed query from a short one is being lied to by the readout itself. DELIBERATELY CONSERVATIVE — vendor-prefixed credentials, values assigned to secret-NAMED variables, auth headers, PEM blocks. Things that are secrets by construction. Entropy heuristics and long-opaque-string detection start eating real queries, and a query is evidence: missing an exotic secret costs one redaction nobody made, while eating a query costs the ability to tune the bar at all. The guard pins BOTH directions, and the second half is the one that matters. A scrubber that eats evidence fails silently — it keeps looking like it works while turning the one instrument for tuning a threshold into unreadable stubs, which is the #2663 shape in a new place. So nine REAL queries from this install's near-miss samples must survive byte for byte. If a future pattern touches one, the pattern is too greedy. Verified against the real shapes before commit: six credential formats redacted (fabricated values), nine real queries unchanged, and the payload seam confirmed to store "export API_TOKEN=[redacted:assigned] && git push". This does NOT scrub rows already written. Purging those is separate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 15:38:40 -04:00
bvandeusen	c149ef31a3	wip(394): steps 6+7 — backend path and instruction surfaces	2026-09-11 15:15:33 -04:00
bvandeusenandClaude Opus 5	690ca0306e	feat(rules): the command arm gets its own bar, measured (#3853) ... One threshold served both act arms. The telemetry says they are not the same problem: write_path_rule 2,325 calls, speaks on 37%, near-miss p50 0.6989 pre_tool_rule 11,768 calls, speaks on 2%, near-miss p50 0.6794 The second is not quiet, it is mute — 11,530 of 11,768 calls said nothing, with near-miss p90 at 0.7097 against a 0.72 bar. Refused mass piled one hundredth under the line is what a bar set too high leaves behind, and the note arms are the control: auto_inject refuses at p90 0.5463, write_path at 0.6738, both far below theirs. The cause is query shape, not corpus. A write-path query is a code payload, long and rich — the case 0.72 was calibrated on. A pre-tool query is a shell command, often under a dozen words: less text, less signal, lower scores for the same relevance. MEASURED. Eight replayed queries against the post-#3855 corpus, consequential acts against innocuous ones: 0.7571 git push origin dev consequential 0.7245 cd ...; git fetch; git add -A consequential 0.7193 git pull --rebase origin dev consequential 0.6850 docker compose up -d consequential ------------------------------------- 0.68 0.6735 wc -l src/*.py && date innocuous 0.6544 grep -rn useState src/ innocuous 0.6099 sed -n '120,160p' package.json innocuous 0.6056 ls -la && cat README.md innocuous At 0.72 three of four consequential acts retrieved nothing, including `git pull --rebase origin dev`, where rules 153, 1 and 2 all ranked correctly between 0.7126 and 0.7193 and were all refused. The separation is 0.0115 wide. That is a direction, not a settled number, and the comment says so — near_miss_samples on a few days of post-#3855 traffic is what settles it. This also corrects an assumption the old comment stated: it argued 0.68 sat "below where this corpus's noise sits", inferring a higher floor from the corpus being homogeneous. Measured, the command arm's noise ceiling is 0.6735, so 0.68 clears it barely rather than sitting under it. Lowering is safer now than it would have been. Until #3851 this arm had one slot, so the bar was the only noise control; the band now filters downstream, so the bar's job shrank and the bar can. write_path_rule is unchanged — healthy at 0.72 on its own evidence. Guards: the two bars parse independently, garbage falls back to its OWN default rather than to the sibling's (which would silently re-merge them), the command default stays below the write-path default as a direction check, and each arm both SEARCHES and REPORTS at its own bar. That last one is a failure the single-bar code could not have had: retrieval_logs.threshold is what near-miss analysis is read against, so an arm searching at one number and logging another misreports the refusal and invites moving the bar that was already right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 14:36:18 -04:00
bvandeusenandClaude Opus 5	40189147d2	fix(rules): a shortened rule line must not decide what it says about holding (#3851) ... CI run 6485 was red. Six failures, three causes, and only one of them was a stale test. THE REAL DEFECT. The compact branch dropped the `seen` TAIL along with the trigger, so a rule the session had already been told rendered exactly like one it had not. #3750's whole argument is that those are different claims — a repeat is rendered precisely because the session may no longer HOLD what it was told — and the tail is the entire difference a reader can act on. test_a_rule_the_session_already_holds_is_referenced_not_re_offered caught it within one commit, which is that guard working as intended. Fixed by keeping the tail and dropping only the trigger, which is both the cheaper and the safer cut: a trigger runs 300-400 characters after #3855, a tail about 100. Re-measured on the real renderer — top-full-plus-references is ~299 tokens against ~568 for five full lines, so about 2x the old single line rather than the 1.4x claimed before, for four more rules and no lost information. The comments carrying the old figure are corrected rather than left to read as a decision nobody made. THE FIXTURE THAT STRADDLED THE BAND. `_THREE_HITS` spanned 0.81-0.74 against a 0.05 band, so the act arms dropped its lowest hit and four cases of test_both_recorders_report_the_same_rules_for_one_call failed reporting a count mismatch — under a message blaming the exclusion filter. A guard pointing confidently at the wrong subsystem costs more than no guard, because it is believed. Scores retightened to 0.81/0.80/0.79 and the precondition is now asserted by a named test, so a future band change is told where the problem is instead of through four confusing failures. THE STALE CONSTANT GUARD. test_the_rule_arm_asks_for_one_rule_not_two pinned RULEHINT_LIMIT == 1 — a real decision, correctly guarded, for a world with a resident set. Rewritten to pin what replaced it, as relationships rather than values (rule 115): the arm can return several, and rules are narrowed HARDER than the notes menu because they measured flatter, not sharper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 14:07:32 -04:00
bvandeusenandClaude Opus 5	10343a6019	feat(rules): an act surfaces a banded SET of rules, quieter after the first (#3851) ... RULEHINT_LIMIT was 1. That was correct while retrieval SUPPLEMENTED a 33-rule resident set — one salient rule beside everything already loaded. Milestone 394 removes residency, and then this arm is the whole delivery: `git push origin dev` is governed by rules 1, 2, 9 and 140 simultaneously, and each alone permits the mistake the others catch. A cap plus a band, not a bigger cap. The old argument's real content is that a fixed k invents lines — it fills slots whether or not anything deserves them. A band keeps only what scored close to the top, so one clearly relevant rule still shows one and four competing rules show four. The corpus decides; the cap is a ceiling on the worst case, not the usual answer. MEASURED, AND IT CORRECTED THE PREDICTION. The expectation was that rules would rank sharply, since rule_document() shapes them like snippets and note 2485 measured snippets separating their top hit by 0.153 against 0.010-0.023 for every other kind. Three probes against real act queries say otherwise: `git push origin dev` top 0.757, gap 0.022 `docker compose up -d` top 0.685, gap 0.016 a bare-owner-filter query top 0.656, gap 0.020 Dev-log territory, not snippet territory — rules arrive as a packed block, so shaping alone did not buy separation. The band is therefore narrow: at 0.10 (the notes menu's value) every one of the top eight on the push probe falls inside, including a CI-registry rule and another project's branch policy. 0.05 admits about three ranks. COST, MEASURED RATHER THAN ASSUMED. The old comment claimed a line costs ~40 tokens. It is ~143 once the trigger is rendered, and #3855 roughly tripled trigger lengths, so five full lines are ~646 tokens before EVERY Bash call. Hence rank decides volume: the top hit keeps the full rendering, later hits are cited (~198 tokens total, 1.4x the old single line, for four more rules). The old paragraph's instinct — a fourth voice at full volume is where a reader stops reading — is answered by making later lines quieter rather than by refusing to have them. Band before dedup, deliberately. The band is a statement about scores; letting the ledger reorder it would make "you were told this already" change what counts as relevant. Same axis independence the renderer already keeps between `kind` and `seen`, and `rule_ids` stays fresh-only (#3752) so #3668's identity between logged results and surfacing rows survives. `suppressed` now covers both causes and says so. Both act arms take the same band: their score distributions are the same shape, and only the query differs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 14:00:36 -04:00
bvandeusenandClaude Opus 5	8e06cdf749	feat(rules): the trigger contract is shown as a worked contrast, and pinned (#3855) ... Follows 8c9f947, which taught the trigger shape on the two update_* surfaces but left the softer regression unguarded: guidance kept and abstracted back to "name the moment in session vocabulary" — advice about being concrete that is not itself concrete, which is the shape that was already on file while the corpus filled with categories. Two attempts to detect that in free prose were written and discarded: - Counting quoted multi-word phrases anywhere in a docstring measured ambient quotation rather than demonstrated triggers. It PASSED the abstracted version by scoring unrelated prose, and text with an odd number of quote characters produced matches spanning the gap BETWEEN two unrelated phrases. - Scoping that count to a window after each trigger mention then FAILED create_preference in its CORRECT state, its examples sitting further from the first mention than any defensible window reaches. Both were proxies inferring demonstration from prose. Where a property cannot be measured, changing the shape of the thing is cheaper than a cleverer measurement — so all five trigger-writing surfaces now carry a two-line labelled contrast: RETRIEVES: "the migration failed with a check violation on a column we just extended" COLLAPSES: "when working on migrations" Unambiguous to parse, free in its wording, and a better teaching form than the sentences it replaces: the labels name the mechanism, so they do work for the reader rather than only for the test. The guard now pins both halves — the field is documented, and the contrast is present, complete and non-identical. Falsified against three regressions before committing: the paragraph stripped, the examples abstracted away, and one half of the pair removed. All three fail; the current tree passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 11:48:14 -04:00
bvandeusenandClaude Opus 5	8c9f947f09	feat(rules): the update surfaces teach the trigger shape, not just the create ones (#3855) ... A trigger is two-thirds of a rule's embedded document, so one naming a CATEGORY rather than a moment collapses the record toward its title and it never arrives. #3835 measured that across 113 rules; #3855 hit it again on the eight preferences, where six named a category and two did not. The split was not carelessness, it was an uneven contract. create_rule has carried the full argument since c61925b (2026-08-27) and the two preferences authored that day got good triggers; the six written weeks earlier got categories. The guidance worked wherever it existed — and it existed on three of five write surfaces. Both update_* tools were silent, and the update path is where every RETROFITTED trigger is written, which is most of them: a trigger that already reads fine as English is the one nobody rewrites. So: - update_rule gains the retrofit case, which is a different trap from the create case. There the field is empty and the instruction is "write one". Here one exists, reads perfectly well, and the honest-looking verdict is that it is fine. - update_preference gains it too, plus why the field is load-bearing there specifically: preferences get a reserved slot filled by a kind-filtered query at limit=1, so the corpus ranks against ITSELF and the trigger is nearly all that separates one from the next. - create_preference and create_project_rule now SHOW a moment instead of describing one. Advice about being concrete that is not itself concrete is the shape that was already on file while the corpus filled up. The guard pins one property: a tool taking when_to_apply mentions it. That is exactly what update_preference failed. The surface list is derived from register() rather than hand-kept, so a write tool added later is in scope the day it lands. Two stronger predicates were written for the softer regression — guidance kept but abstracted — and both were discarded after falsification: counting quoted phrases measured ambient quotation and passed the broken version, and scoping that count to a window failed create_preference while correct. Rule 167 settles it; the discarded attempts are recorded in the test docstring so the next author does not repeat them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-11 11:39:00 -04:00