feat(retrieval): a System's charter becomes an answer, not just a filter (#4251)

Step 2 of #4251. A System's `description` is a charter — several hundred words
saying what belongs in that area and what does not — and it is the answer to
"which part of this codebase does X live in". There was no semantic path to
one: `list_systems` enumerates, and `search(system_id=…)` uses a System as a
FILTER over notes. So a System could narrow a search and could never be the
answer to one, and an agent asking where a record belonged had to read every
charter or guess.

ITS OWN SEARCH, not a `content_type` over notes, for the reason note 3163
gives about milestones: the row could be shared, the search cannot. A charter
competing with the whole note corpus for one top-k is outranked by the records
filed under it — the right answer crowded out by its own contents — and "where
does this belong?" is a different question from "what prior art is there?",
which a caller asking one should not have to read past answers to.

So `system_embeddings` (0107) joins note_, rule_ and milestone_embeddings as
the fourth sibling, with `system_document`, `upsert_system_embedding`,
`semantic_search_systems`, a startup backfill and `search(content_type=
"system")`. Scoped like milestones: with a project_id, that project's Systems
if the caller can read the project (rule 78); without one, the caller's own.
Archived Systems are excluded — an archived area is one the operator has said
is no longer where things go, which is exactly the question being asked.

`system_document` is the plainest of the four shapes on purpose. A charter is
already written as the thing this search has to match, in the words someone
asking would use — so there is no trigger to synthesise as `rule_document`
must, and no second record to gather as `task_document` must. The stored
charter IS the sharp document, the way a snippet's is. `color`, `status` and
`order_index` stay out: presentation and bookkeeping, and a vector carrying
them would be answering a question nobody asks of a charter.

The search publishes `report["best_chunk"]` from the start rather than being
retrofitted, which is what #4251 asked of any fourth search. It matters more
here than anywhere: a charter runs long and a result shows its NAME, so a match
on the paragraph that actually decides where a record belongs would otherwise
be previewed by two words that cannot say.

The id that comes back is the one `system_id`, `system_ids` and
`list_system_records` already take, so the answer to "where does this belong?"
is directly usable as "show me what is there" and as "file it here".

`embed_system` sits beside `notes.embed_note` at the service for #2056's
reason — every door gets it by construction. Not called on delete: that is a
soft delete and the search joins through `System`, so the vectors are already
unreachable, and leaving them means a restore is findable again immediately.
`system_embeddings` is declared in backup's `_NOT_INCLUDED` as derived, beside
its three siblings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/mcp/tools/search.py

@@ -15,7 +15,7 @@ from scribe.services.knowledge import content_type_filters
 from scribe.services.text import MATCHED_PASSAGE, excerpt_fields
 from scribe.services.embeddings import (
     DEFAULT_SIMILARITY_THRESHOLD, semantic_search_milestones, semantic_search_notes,
-    semantic_search_rules,
+    semantic_search_rules, semantic_search_systems,
 )
 from scribe.services import rulebooks as rulebooks_svc
 from scribe.services.retrieval_telemetry import record_retrieval, retrieval_summary
@@ -38,7 +38,7 @@ _EXCERPT_CHARS = 1000
 # These two do not reach `semantic_search_notes` at all: they have their own
 # search and their own result shape, so they are dispatched before the mapping
 # and passed in only so the refusal message lists everything THIS door takes.
-_OWN_SEARCH = ("rule", "milestone")
+_OWN_SEARCH = ("rule", "milestone", "system")
 
 
 async def _search_rules(uid: int, q: str, limit: int, project_id: int) -> dict:
@@ -141,6 +141,53 @@ async def _search_milestones(uid: int, q: str, limit: int, project_id: int) -> d
     }
 
 
+async def _search_systems(uid: int, q: str, limit: int, project_id: int) -> dict:
+    """Systems by meaning — "where does this belong?" (#4251).
+
+    A System's `description` is a charter: several hundred words saying what
+    belongs in that area and what does not. `list_systems` enumerates them and
+    `system_id` filters by one, so before this a System could NARROW a search
+    and could never be the answer to one — an agent asking where a record
+    belonged had to read every charter or guess.
+
+    Its own result shape and its own search, not a `content_type` over notes,
+    because the question is different: "where does this belong?" is not "what
+    prior art is there?". A charter competing with the whole note corpus for
+    one top-k would also be outranked by the records filed under it, and the
+    right answer would be crowded out by its own contents.
+
+    The charter's `matched` passage comes along rather than the whole thing.
+    A charter is long and the paragraph that decides where a record belongs is
+    the one worth reading; the rest is get_system (#4243).
+
+    The id that comes back is the one `search(system_id=…)`,
+    `list_system_records` and every `system_ids` argument take — so the answer
+    to "where does this belong?" is directly usable as "show me what is there"
+    and as "file it here".
+    """
+    report: dict = {}
+    raw = await semantic_search_systems(
+        uid, q, project_id=project_id or None, limit=limit, report=report,
+    )
+    chunks = report.get("best_chunk") or {}
+    return {
+        "results": [
+            {
+                "id": sys_.id,
+                "name": sys_.name,
+                **excerpt_fields(
+                    sys_.description or "", chunks.get(int(sys_.id)),
+                    _EXCERPT_CHARS, key="matched",
+                ),
+                "project_id": sys_.project_id,
+                "similarity": float(score),
+            }
+            for score, sys_ in raw
+        ],
+        "total": len(raw),
+    }
+
+
 def result_excerpt(note, chunk: dict | None) -> dict:
     """The part of a record a caller judges "should I open this?" on.
 
@@ -204,8 +251,14 @@ async def search(
             claim that the corpus holds nothing, and a typo must not be able
             to make that claim.
 
-            Or 'rule' (RULES only — the operator's standing
-            instructions, searchable by meaning since milestone 307).
+            THREE KINDS WITH THEIR OWN SEARCH AND THEIR OWN RESULT SHAPE,
+            because each answers a question no note search can:
+            'rule' (the operator's standing instructions — searchable by
+            meaning since milestone 307), 'milestone' ("is there already a
+            plan for this?"), and 'system' ("where does this belong?" — a
+            System's charter says what belongs in an area and what does not,
+            and the id it returns is the one `system_id` and `system_ids`
+            take).
             Reach for 'rule' when you want to know whether a standing
             instruction covers something: "is there a rule about release
             tagging?". A hit carries the rule's `why` and `how_to_apply`,
@@ -258,6 +311,8 @@ async def search(
         return await _search_rules(uid, q, limit, project_id)
     if content_type == "milestone":
         return await _search_milestones(uid, q, limit, project_id)
+    if content_type == "system":
+        return await _search_systems(uid, q, limit, project_id)
     filters = content_type_filters(content_type, extra=_OWN_SEARCH)
     is_task = filters.get("is_task")
     t0 = time.perf_counter()