feat(retrieval): a System's charter becomes an answer, not just a filter (#4251)

Step 2 of #4251. A System's `description` is a charter — several hundred words
saying what belongs in that area and what does not — and it is the answer to
"which part of this codebase does X live in". There was no semantic path to
one: `list_systems` enumerates, and `search(system_id=…)` uses a System as a
FILTER over notes. So a System could narrow a search and could never be the
answer to one, and an agent asking where a record belonged had to read every
charter or guess.

ITS OWN SEARCH, not a `content_type` over notes, for the reason note 3163
gives about milestones: the row could be shared, the search cannot. A charter
competing with the whole note corpus for one top-k is outranked by the records
filed under it — the right answer crowded out by its own contents — and "where
does this belong?" is a different question from "what prior art is there?",
which a caller asking one should not have to read past answers to.

So `system_embeddings` (0107) joins note_, rule_ and milestone_embeddings as
the fourth sibling, with `system_document`, `upsert_system_embedding`,
`semantic_search_systems`, a startup backfill and `search(content_type=
"system")`. Scoped like milestones: with a project_id, that project's Systems
if the caller can read the project (rule 78); without one, the caller's own.
Archived Systems are excluded — an archived area is one the operator has said
is no longer where things go, which is exactly the question being asked.

`system_document` is the plainest of the four shapes on purpose. A charter is
already written as the thing this search has to match, in the words someone
asking would use — so there is no trigger to synthesise as `rule_document`
must, and no second record to gather as `task_document` must. The stored
charter IS the sharp document, the way a snippet's is. `color`, `status` and
`order_index` stay out: presentation and bookkeeping, and a vector carrying
them would be answering a question nobody asks of a charter.

The search publishes `report["best_chunk"]` from the start rather than being
retrofitted, which is what #4251 asked of any fourth search. It matters more
here than anywhere: a charter runs long and a result shows its NAME, so a match
on the paragraph that actually decides where a record belongs would otherwise
be previewed by two words that cannot say.

The id that comes back is the one `system_id`, `system_ids` and
`list_system_records` already take, so the answer to "where does this belong?"
is directly usable as "show me what is there" and as "file it here".

`embed_system` sits beside `notes.embed_note` at the service for #2056's
reason — every door gets it by construction. Not called on delete: that is a
soft delete and the search joins through `System`, so the vectors are already
unreachable, and leaving them means a restore is findable again immediately.
`system_embeddings` is declared in backup's `_NOT_INCLUDED` as derived, beside
its three siblings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/embeddings.py

@@ -29,6 +29,7 @@ from scribe.services.access import can_read_project, notes_visibility_clause
 if TYPE_CHECKING:  # resolves forward refs without importing at runtime
     from scribe.models.milestone import Milestone
     from scribe.models.rulebook import Rule
+    from scribe.models.system import System
 
 logger = logging.getLogger(__name__)
 
@@ -1451,6 +1452,211 @@ async def semantic_search_milestones(
     return kept
 
 
+# --- Systems: the charter as an ANSWER, not a filter (#4251) -----------------
+
+
+def system_document(
+    name: str | None, description: str | None
+) -> tuple[str | None, str | None]:
+    """The (title, body) a System is EMBEDDED as — its name and its charter.
+
+    The plainest of the four document shapes, and deliberately so. A System's
+    `description` is already written as the thing this search has to match: it
+    says what belongs in the area and what does not, which is the answer to
+    "where does this go?" in the words someone asking would use. There is no
+    trigger to synthesise, as `rule_document` must, and no separate record to
+    gather in, as `task_document` must — the stored charter IS the sharp
+    document, the way a snippet's is.
+
+    `color`, `status` and `order_index` are left out. They are presentation and
+    bookkeeping; a vector that carried them would be answering a question
+    nobody asks of a charter.
+
+    A System with no charter yet degrades to its name. It still embeds, just
+    weakly — a bare name is exactly what the model docstring says is never
+    enough, and this is an argument for writing the charter, not for padding
+    the document with whatever is to hand.
+    """
+    return (name or "").strip() or None, (description or "").strip() or None
+
+
+async def upsert_system_embedding(
+    system_id: int, name: str | None, description: str | None
+) -> None:
+    """Chunk, embed and persist a System's vectors. Safe to fire-and-forget.
+
+    The third sibling's contract exactly: one document definition shared by the
+    write path and the backfill, and an atomic per-System replacement guarded
+    by the parent-row claim (#3262), so a System deleted mid-refresh wins.
+    """
+    from scribe.models.embedding import SystemEmbedding
+    from scribe.models.system import System
+
+    doc_title, doc_body = system_document(name, description)
+    chunks = chunk_document(doc_title, doc_body)
+    try:
+        if not chunks:
+            async with async_session() as session:
+                await session.execute(
+                    delete(SystemEmbedding).where(SystemEmbedding.system_id == system_id)
+                )
+                await session.commit()
+            return
+    except Exception:
+        logger.warning("Failed to clear embedding for system %d", system_id, exc_info=True)
+        return
+
+    try:
+        vectors = await get_embeddings(chunks)
+    except Exception:
+        logger.debug("Skipping embedding for system %d — embedder unavailable", system_id)
+        return
+
+    try:
+        async with async_session() as session:
+            if not await _claim_parent_row(session, System.id, system_id, "system"):
+                return
+            await session.execute(
+                delete(SystemEmbedding).where(SystemEmbedding.system_id == system_id)
+            )
+            for index, (chunk, vector) in enumerate(zip(chunks, vectors)):
+                session.add(SystemEmbedding(
+                    system_id=system_id, chunk_index=index, embedding=vector,
+                    chunk_text=chunk, chunker_version=CHUNKER_VERSION,
+                ))
+            await session.commit()
+    except Exception:
+        logger.warning("Failed to persist embedding for system %d", system_id, exc_info=True)
+
+
+async def semantic_search_systems(
+    user_id: int,
+    query: str,
+    *,
+    project_id: int | None = None,
+    limit: int = 5,
+    threshold: float = _SIMILARITY_THRESHOLD,
+    report: dict | None = None,
+) -> list[tuple[float, "System"]]:
+    """Return up to *limit* (score, system) pairs most like *query*.
+
+    Answers "where does this belong?" — the question asked before filing a
+    record or opening a file, and the one that had no tool: `list_systems`
+    enumerates and `system_id` filters, so a charter could narrow a search and
+    could never be the answer to one.
+
+    ITS OWN SEARCH rather than a note kind, for what note 3163 says about
+    milestones: a charter competing with the whole note corpus for one top-k
+    would be outranked by the records filed under it, and the right answer
+    would be crowded out by its own contents. The questions are different too —
+    "where does this belong?" is not "what prior art is there?" — and a caller
+    asking one should not have to read past answers to the other.
+
+    SCOPE. With `project_id`, that project's Systems, provided the caller can
+    read the project (access.can_read_project, rule 78) — a collaborator on a
+    shared project sees its areas, which is the point of a charter. Without
+    one, the Systems the caller owns across their projects. Archived Systems
+    are excluded: an archived area is one the operator has said is no longer
+    where things go, and that is exactly the question being asked.
+
+    Collapses to best-chunk-per-System and publishes the winning chunk, like
+    the sibling searches — from the start rather than retrofitted (#4243).
+    Returns an empty list if the embedder is unavailable, the project is not
+    readable, or on any error: a recall aid must never break the call it
+    serves.
+    """
+    from scribe.models.embedding import SystemEmbedding
+    from scribe.models.system import System
+
+    if not query or not query.strip():
+        return []
+    try:
+        query_vec = await get_embedding(query)
+    except Exception:
+        logger.debug("System search skipped — embedder unavailable")
+        return []
+
+    distance = SystemEmbedding.embedding.cosine_distance(query_vec)
+    try:
+        if project_id:
+            if not await can_read_project(user_id, project_id):
+                return []
+            scope = System.project_id == project_id
+        else:
+            scope = System.user_id == user_id
+        async with async_session() as session:
+            rows = (await session.execute(
+                select(
+                    System,
+                    distance.label("distance"),
+                    SystemEmbedding.chunk_index,
+                    SystemEmbedding.chunk_text,
+                )
+                .select_from(SystemEmbedding)
+                .join(System, SystemEmbedding.system_id == System.id)
+                .where(
+                    scope,
+                    System.deleted_at.is_(None),
+                    System.status != "archived",
+                )
+                .order_by(distance)
+                .limit(limit * _CHUNK_OVERFETCH)
+            )).all()
+    except Exception:
+        logger.warning("System semantic search failed", exc_info=True)
+        return []
+
+    best: dict[int, tuple[float, object]] = {}
+    # A charter runs to several hundred words and a result shows its NAME — so
+    # a match on the paragraph that actually decides where a record belongs
+    # would be previewed by two words that cannot. The winning chunk is what
+    # the caller should see (#4243).
+    won: dict[int, dict] = {}
+    for system, dist, chunk_index, chunk_text in rows:
+        score = 1.0 - float(dist)
+        if system.id not in best or score > best[system.id][0]:
+            best[system.id] = (score, system)
+            won[int(system.id)] = {
+                "index": int(chunk_index), "text": chunk_text or "",
+            }
+    ranked = sorted(best.values(), key=lambda pair: pair[0], reverse=True)
+    kept = [pair for pair in ranked if pair[0] >= threshold][:limit]
+    record_best_chunk(report, {
+        int(sys_.id): won[int(sys_.id)] for _s, sys_ in kept if int(sys_.id) in won
+    })
+    return kept
+
+
+async def backfill_system_embeddings() -> None:
+    """Embed Systems that have no current vectors. Runs at startup beside the
+    note, rule and milestone backfills; a CHUNKER_VERSION bump re-embeds.
+
+    This is also the pass that makes every existing charter findable at all —
+    Systems got vectors in #4251, so before it runs there are none."""
+    from scribe.models.embedding import SystemEmbedding
+    from scribe.models.system import System
+
+    try:
+        async with async_session() as session:
+            current = select(SystemEmbedding.system_id).where(
+                SystemEmbedding.chunker_version == CHUNKER_VERSION
+            )
+            stale = (await session.execute(
+                select(System.id, System.name, System.description)
+                .where(System.deleted_at.is_(None), System.id.notin_(current))
+            )).all()
+    except Exception:
+        logger.warning("System embedding backfill: failed to query systems", exc_info=True)
+        return
+
+    if not stale:
+        logger.info("System embedding backfill: all systems current at chunker v%d", CHUNKER_VERSION)
+        return
+    logger.info("System embedding backfill: embedding %d system(s)", len(stale))
+    for system_id, name, description in stale:
+        await upsert_system_embedding(system_id, name, description)
+
+
 async def backfill_milestone_embeddings() -> None:
     """Embed milestones that have no current vectors. Runs at startup beside
     the note and rule backfills; a CHUNKER_VERSION bump re-embeds."""