Rules become findable: canon catalog, triggers, tiers, edges, retrieval, surfacing (milestone 307, steps 1–5) #131

Merged

bvandeusen merged 13 commits from dev into main 2026-08-26 17:12:40 -04:00

Conversation 0 Commits 13 Files Changed 50

+3824 -220

13 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	02c1e37620	feat(rules): the write path can notice a standing rule it was never given (#3031, milestone 307 step 5, hook arm) ... A conditional rule is not resident, so a session can be about to violate one it was never handed. This arm notices: when what is being written resembles a rule's trigger, the hint names it and says to read it before deciding it does not apply. A SUGGESTION, and the plan was wrong about why it could be more. It claimed the hook "already resolves a path to an area" — it does not, and nothing in Scribe maps a path to a System or a canonical area (build_write_path_hint resolves paths against snippet LOCATIONS, a different index; the learned-alias idea belongs to another project). Correction logged on the task. Rather than invent path→area inference to make a stale claim true, the arm does what D7 already decided and what this surface already IS: tags bind at enter_project, meaning suggests here. The header of the hook says NEVER BLOCKS; dressing a hint up as binding would have been the actual mistake. CONDITIONAL RULES ONLY. An always-on rule is already in the session, so re-offering it is noise — and noise on a hint that fires on every write is how a hint gets ignored. Telemetry goes to retrieval_logs, NOT note_usage_events, and that is a correctness call rather than a preference: note_usage ids are REMAPPED on a backup restore, so a rule id written there would come back attached to whatever note took that number — silently corrupting the evidence the next true-up is supposed to read. retrieval_logs is never restored and `source` already separates surfaces. record_retrieval's `results` type widened to match what it actually needs (an `.id`), instead of passing a Rule to something annotated Note. The rule dedup gets its OWN state file and query parameter, like the three channels before it — #2708's lesson was that one shared channel lets a hint of one class silence a different class that had never been shown. Plugin version bumped: a hook change clients cannot see did not ship (#1040). The stub is autouse in conftest rather than added to forty-odd call sites: the arm loads an embedding model, and every existing test that stubs the NOTES search would otherwise pull a real model in through the one arm it had no way to know about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 16:35:36 -04:00
bvandeusenandClaude Opus 5	4585cda3ff	fix(rules): import or_ in rulebooks — the tier gate used it unimported (#3031) ... NameError at runtime: the tier/area clause and co_surfaced_partners both build an or_(), which rulebooks.py never imported. py_compile passes on this (it is a name error, not a syntax error) and so did the whole unit suite. Where it surfaced is the useful part. The unit tests mock the session, so the project-areas query returns empty, `reachable` stays None, and the or_ branch is never taken — they exercised the path that avoids the bug. Only the integration lane, with a real project and real rows, went down the branch that needed the name. Six integration tests caught it, including the inception one that merely calls get_applicable_rules in passing. A reminder about which lane proves what: mocking the thing that selects the branch means the branch is untested by construction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 15:13:40 -04:00
bvandeusenandClaude Opus 5	cd9aa87aa4	feat(rules): tier 1 preloads, tier 2 arrives by area — and a split rule can no longer be read half-way (#3031, milestone 307 step 5) ... The payoff step: a rule stops having to be resident to be honoured. - list_always_on_rules returns the ALWAYS-ON tier only. It is the session-start call, made before any project is in scope, so there is no area vocabulary to match a conditional rule against yet. - get_applicable_rules carries a conditional rule when the project works in an area the rule is tagged to — resolved through systems.canonical_id, so the project's own NAME for the area is irrelevant, which is the entire reason the catalog exists. The gate is applied IN SQL, so `limit` counts rules that will actually surface rather than rules about to be dropped. - Bindingness is a deterministic TAG match, never a similarity score (D7). The vector channel stays a suggestion, in search. co_surfaced_partners is the fix that rule 144 never had. It was split off rule 46 and folded back the same day because "either rule could surface without the other and miss exposing a project to what the entire shape is intended to be" — correct, and merging was the only remedy available. Now a partner ARRIVES with its other half even when nothing else selected it, tagged `via: co_surfaces` so the payload says why. Two limits, both deliberate: only rules the caller owns, because an edge is not a back door into someone else's rulebook; and a project's suppressions are passed as exclusions, because an explicit mute is a decision and an edge does not outrank it. COMPATIBILITY, asserted first in the integration test rather than reasoned about: a rule with no tier, no areas and no edges binds exactly as it did before any of this existed. `tier` defaults to always_on, so an install upgrades and every rule it already had keeps arriving. Getting that backwards would silently stop enforcing rules people rely on, which is worse than any amount of payload bloat. Four unit tests were coupled to the ORDER of a mocked session's execute() calls, so a new query broke them. Rather than pad the sequence and deepen that coupling, the three post-query lookups are stubbed by name — they have their own coverage, and the real wiring is proven against Postgres. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 15:10:41 -04:00
bvandeusenandClaude Opus 5	6ada97bb0b	fix(rules): resolve the Rule forward ref for ruff and readers (#3030) ... F821: the quoted `"Rule"` in semantic_search_rules' return annotation evaluates fine at runtime — a string inside a subscript is a value, not a name lookup — but it points at nothing a reader or a type checker can follow, which is what ruff is objecting to and it is right to. A TYPE_CHECKING import resolves it at zero runtime cost, while the real import stays inside the function so this module still does not pull in the rulebook models. Placed after the import block, where isort wants the guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 15:03:40 -04:00
bvandeusenandClaude Opus 5	95a37318fc	feat(rules): rules become findable by meaning (#3030, milestone 307 step 4) ... Rules were the only major record type with no vector, so `search` could never return one and a rule could arrive only by being preloaded. That single fact is what made every rule compete for one always-on budget. THE DECISION THE TASK ASKED FOR, made explicitly: a sibling rule_embeddings table, not a polymorphic embedding row. The ROW could have been generalised; the SEARCH could not. semantic_search_notes is Note-specific scoping end to end — the visibility clause, the supersession penalty, note_type/task_kind/system filters — and a rule shares none of it, scoping instead by rulebook ownership or project. Generalising the row while still needing two searches is the worst of both: a key with referential integrity to neither table, on the path every session start runs, to share four columns. What is genuinely common is BEHAVIOUR — get_embedding, chunk_document, embedding_text, CHUNKER_VERSION — and those are reused as-is. Sharing them is the DRY win; sharing the table would have been the DRY costume. The document shape is measured, not chosen (note 2485). That pass found the snippet was the only discriminative record in the corpus — a 0.153 top-to-second gap against 0.010-0.023 — and that the cause was its SHAPE: purpose stated twice in a short single-topic document. rule_document reproduces it: the trigger in the title AND as the body's first line. And it excludes `why`, which matters more than any of it. `why` is dated incident narrative — rule 46's runs to 4,300 characters — and long multi-topic prose is exactly what made sixteen dev-logs mutually indistinguishable. Adding it would not give the vector more to work with; it would give every rule the SAME thing to work with. rule_document takes no `why` parameter at all, so a well-meaning caller cannot pass one. A rule with no trigger degrades to title + statement — findable, less sharp. That is an argument for backfilling triggers (step 6), not for padding the document with whatever text is nearby. search(content_type="rule") returns the rule WITH its why and how_to_apply: they are its operational half, the session payload never carries them, and a caller who went looking should not have to re-fetch. Writes re-index fire-and-forget like notes; startup backfills in its own try block so neither backfill can skip the other. rule_embeddings is derived, so it joins note_embeddings in the backup's explicitly-NOT-included list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 15:00:49 -04:00
bvandeusenandClaude Opus 5	682bea5257	fix(rules): the third row literal — fetchRules builds a list row too (#3029) ... vue-tsc caught what I missed: there were THREE places hand-building a rule list row, not two. fetchRules mapped full rules down to the same four fields in a spot far from the other two, so consolidating the pair I could see left this one behind — which is precisely how the server side ended up with three divergent trim dicts in the first place. All three now go through toHeader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 14:26:20 -04:00
bvandeusenandClaude Opus 5	8b60d552d2	feat(rules): the rule editor asks when it applies, and the list shows its age (#3029, milestone 307 step 3, UI) ... Rule 27 — the schema and both doors shipped with no human surface, so step 3 was not shippable until this. RuleEditorSlideOver gains the trigger, the tier, the areas, and a read-only view of the rule's edges. The tier is a radio pair carrying the test itself rather than a bare toggle: can you name the trigger WITHOUT naming a system, an artifact type or a moment? If the honest answer is "whenever you are working", it is always on. It also says why conditional is not a demotion — it costs nothing when irrelevant, which is what lets a rule be as long as it needs to be. The relations block states the rule the whole milestone turns on: rules that FAIL TOGETHER are linked, never merged. RuleListPane shows the trigger and the LAST-CHANGED DATE on every row, and marks conditional only — always_on is the default and badging every row would say nothing. The date is the cheap triage the FabledCurator case wanted: a rule whose age predates the capability it duplicates is visible at a glance instead of needing a get_rule to find out. ProjectRulesTab's inline create form gains the same two fields, because a project rule bloats exactly the way a family one does — FabledCurator has 23 of them. Two type fixes the new shapes forced, both worth keeping: - toHeader() in the store: a list row is the server's rule_brief, so patching a list locally has to mirror every field it carries or the two disagree. There were two hand-built four-field literals doing that job. - ApplicableRules.rules / .project_rules are now described AS RuleHeader rather than as two more hand-written shapes — the same builder produces them, so the same type should describe them. groupByRulebookAndTopic skips a null-topic rule rather than widening TopicGroup to accept one: a rule carries topic_id XOR project_id, so a null topic in that list means something is wrong upstream, and a widened type would hide it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 14:23:40 -04:00
bvandeusenandClaude Opus 5	ffb7a0fe38	feat(rules): both doors carry the trigger, the tier, the areas and the edges (#3029, milestone 307 step 3, surfaces) ... MCP and REST both gain when_to_apply / tier / system_ids / arose_from_id on create and update, plus relate_rules / unrelate_rules for the typed edges, and get_rule now returns a rule's areas and relations alongside it. rule_detail() is a SERVICE function, not one per door. It started as a copy in each — identical, and the prior-art hook flagged it immediately, which is the same lesson rules_payload (#2858) already recorded: a second copy drifts. Both doors call the one seam, so create, update and get cannot disagree about what a rule looks like coming back. The authoring guidance lands in create_rule's docstring rather than in a rule, per rule 119 as the operator described it: this is behaviour every instance should inherit, not one operator's preference. It states the test — ONE RULE = ONE THING YOU COULD VIOLATE. Rules that FAIL TOGETHER get linked with relate_rules(kind="co_surfaces"), never merged into one row. — and names why merging loses: a merged rule cannot be cited, surfaced or suppressed a clause at a time, and it grows without limit because adding to it is always cheaper than adding a rule. create_project_rule says the same about "overrides", which is what FabledCurator's 85/86 should have been instead of near-copies that drift from their parent. The tier arg carries the test itself: can you name the trigger WITHOUT naming a system, an artifact type or a moment? If the honest answer is "whenever you are working", it is always_on. Tests: the applicable-rules cases fabricated raw tuples matching the old column lists, so they move to the entity shape via fake_rule; new cases pin rule_brief (a DATE not a stamp, the depth left to get_rule, no null keys) and that an unknown tier falls back to BINDING. fake_rule gains when_to_apply / tier / arose_from_id for the note-2109 reason the helper exists: unnamed, they would be truthy MagicMocks. The tool tests stub the new rule_detail seam — they are about argument forwarding and have no database. The module header's "Sixteen tools" had been wrong for two milestones; the registration count test is what actually catches that, so the header now says so instead of carrying a number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 14:17:41 -04:00
bvandeusenandClaude Opus 5	6ddb8bf859	feat(rules): a rule can say when it applies, which area it is about, and what it belongs with (#3029, milestone 307 step 3, schema) ... A rule could not state its trigger, its area, or its siblings, so all three were being written as prose instead: a System's charter restating rule text, a `why` naming the note that caused it, and two halves of one shape merged into a single row because either could surface without the other. Migration 0088 adds the four fields those workarounds stood in for: - `when_to_apply` — the trigger. Nullable in the DB and required at the service layer: existing rules have none and a migration cannot invent one. - `tier` — always_on | conditional, defaulting to always_on. This migration therefore changes NOTHING about which rules bind; an install upgrades and every rule keeps arriving exactly as before. Getting that backwards is the one failure this milestone exists to prevent, so _valid_tier falls back to always_on rather than silently un-binding a rule with a typo'd tier. - `arose_from_id` — the record that caused the rule, the edge notes and tasks already have. SET NULL: trashing the source does not repeal the rule. - `rule_systems` / `rule_relations` — the canon tag and the typed edges (co_surfaces / overrides / elaborates), each earned from a workaround its absence forced. rule_brief() replaces the THREE hand-written trim dicts that had already diverged — two carried topic_id, one didn't, and none carried the timestamps the model has held all along. That omission is why a rule written before the capability it duplicates was indistinguishable at read time from one still doing work. It now carries updated_at as a DATE: the question is "how old is this", and a full stamp across the always-on set is ~2k characters for precision nobody reads. The two callers select the ENTITY rather than a column list, so rule_brief stays the single place deciding what a surfaced rule says. Backup: both new tables carried, area tags by canonical SLUG (ids are per-install). The rule-relation restore runs after ALL rules exist and after the catalog, because an edge names two rules and a tag names a global row — sections renumbered so the file reads in dependency order. A pre-0088 payload restores with tier=always_on, i.e. binding exactly as when it was taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 14:11:32 -04:00
bvandeusenandClaude Opus 5	67874268bb	test(systems): move the name-gate cases to the service the gate moved into (#3028) ... Two tests further down test_mcp_tool_systems.py still drove the gate through the tool — `svc.list_systems` stubbed, the tool doing the normalising — so the new `await systems_svc.assess_system_name(...)` hit an unstubbed MagicMock. Stubbing them at the tool would have kept testing the wrong layer. The normalisation cases belong with the logic, so they move to tests/test_services_systems.py as real coverage of assess_system_name: case and whitespace folding, exact-beats-overlap (and that an exact hit short-circuits the lesser lookup), no invented match, fail-open on both arms, and silence for a nameless system. What stays the tool's job — rendering a duplicate, applying an exact area, offering an overlap — is already covered at the top of that file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 13:02:09 -04:00
bvandeusenandClaude Opus 5	c58529718b	feat(systems): the catalog reaches the moment a name is minted, and gets a face (#3028, milestone 307 step 2) ... Step 1 found the reason the standard names never held, and it is sharper than "prose doesn't fire": the list WAS real and it WAS seeded — but only on the inception path, for a project with zero Systems. Ad-hoc create_system never consulted it, which is how Forge minted "CI and Release" and Portal minted "CI & release" after the constant already existed. This wires the vocabulary to the moment that mints a name. - services/systems.assess_system_name: the local duplicate gate AND the catalog lookup, in ONE service function both doors call. The gate lived only in the MCP tool, which is exactly how the web UI shipped without a check the agent surface enforced (#2482). REST now answers 409 with the System that already covers the area. - An `exact` catalog hit is APPLIED (mechanical — the names differ only in spelling). An `overlap` is only OFFERED, on both doors: applying a judgment call silently is how a cross-project rule surfaces in the wrong project. - canonical_systems.best_overlap is the ONE scorer behind the create-time offer and the review sweep, so the two surfaces can never name different areas for one System. It also takes the catalog the caller already holds, so the review is not an N+1. UI (folded in from step 1 — rule 27, that step shipped with no human surface): - SystemsSection: a Shared area picker on create and edit, the area on each card, and a collapsed review of proposals that appears only when there is something to decide. `exact` and `overlap` never share a style — one is mechanical, the other is the reviewer's judgment, and presenting them alike is how a wrong mapping gets waved through. - Settings → Admin → Areas: the catalog itself, showing each entry's slug, because the slug is what decides whether two names are the same area and a rename moves it. - A picker rather than a live matcher: reproducing the slug rule in TypeScript would give this feature two matchers to keep in step — the exact drift the catalog exists to end. The server stays authoritative. tests/helpers.fake_system gains canonical_id=None: an unnamed attribute is an auto-MagicMock and therefore truthy, which is the trap that helper exists for (note 2109) and a nullable FK walks straight into it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 12:58:30 -04:00
bvandeusenandClaude Opus 5	879ef3053e	fix(systems): the bootstrap ask reads the catalog, so its test must supply one (#3027) ... CI caught the seam the promotion opened: the standard names now come from an async catalog read, and the unit test patches systems_svc wholesale — so the read raised, the fail-open swallowed it, and the ask shipped without the names it is supposed to carry. Stub standard_systems in that test, and assert the wiring rather than the vocabulary: THAT the seeded set is these eight is migration 0087's business and belongs in the inception integration test, against a real database. Adds the case the promotion actually created — an unreachable or empty catalog must still produce the ask. The names are an aid to the question, not the question; degrading to a weaker nudge is fine, going silent is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 12:34:29 -04:00
bvandeusenandClaude Opus 5	a97547fbc6	feat(systems): the area vocabulary becomes a global table so a rule can point at one (#3027, milestone 307 step 1) ... The eight standard area names already existed — as STANDARD_SYSTEMS, a tuple in services/systems.py that milestone 297 seeds at inception. A constant cannot be a foreign key, so nothing outside a project could reference an area: systems. project_id is NOT NULL, and a rule that spans projects would have to chain itself to one project's row. And because the list only ever applied on the inception-seed path, three spellings of one area reached this instance anyway (CI & runners / CI and Release / CI & release). - canonical_systems: global, no user_id — a shared project inherits the vocabulary instead of re-earning it. Migration 0087 seeds the same eight. - systems.canonical_id: nullable, SET NULL. Association only — no System is renamed and record_systems is untouched, so no record's tags move. - canonical_slug folds &/and, case and punctuation, so spelling variants map mechanically and a real difference ("CI & runners") becomes a proposal a human confirms. propose_mappings reports; set_system_canonical is the only writer. - seed_standard_systems now reads the catalog and maps as it mints, so a project born standard never needs a reconciliation pass. - Catalog writes are admin-only; reads are open — a global list anyone can extend stops being shared. - backup: carried by SLUG, not id (ids are per-install). Restore reuses the target's own rows and only creates entries an admin added on the source; an unknown slug restores unmapped rather than failing. Rule 22: STANDARD_SYSTEMS is removed, not deprecated. Rule 115: nothing seeded names an app, repo or house convention. Design in note 3026. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 12:31:18 -04:00