feat(snippets): reverse lookup — find snippets by repo/path/symbol
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Failing after 27s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 1m42s

"What canonical helpers already live in this file?" was unanswerable:
location lived only in the body markdown. It is now a jsonpath containment
query over the `notes.data` mirror added by migration 0070.

- One predicate in two dialects in services/knowledge.py: SQL (`data @?`,
  applied in the browse arm and the keyword arm before count/pagination, so
  totals stay honest) and Python (`location_matches`, for the semantic arm
  which post-filters candidates it already holds). Both must change together.
- Parts are ANDed within a SINGLE locations entry — repo A in one entry and
  path B in another is not "recorded at A/B". `path` also matches as a
  directory prefix, via jsonpath `starts with` rather than `@>`, which the
  same GIN index serves.
- `repo`/`path`/`symbol` reach the service, the REST list and the MCP tool
  under one name with one default (rule #33); the MCP docstring teaches the
  place form, and so does the reusing-code skill (plugin.json bumped).
- UI: a Location disclosure beside the snippet search, with its own empty
  state — "nothing kept there, so what you're about to write is new."

Settles #2083's open question (pre-0070 NULL `data`) by backfilling after
all: `backfill_snippet_data` runs at startup, deriving the mirror from the
body with the same parser the read path trusts. 0070's caution was about
mangling a hand-edited body; this never touches the body. The alternative
was a permanent second body-regex arm, or a query that silently answers
"nothing here" for an old snippet and gets the helper written twice.

Refs #2083, milestone #232.
This commit is contained in:
2026-07-27 23:09:37 -04:00
parent 0d396de215
commit dd1b5e5ddb
12 changed files with 899 additions and 16 deletions
+9
View File
@@ -170,6 +170,15 @@ def create_app() -> Quart:
await backfill_note_embeddings()
except Exception:
logger.warning("Embedding backfill failed", exc_info=True)
# Snippets written before migration 0070 have no `notes.data` mirror,
# and the location reverse lookup queries that column — an unfilled
# row would read as "no snippet here" rather than as a gap. Separate
# try block so neither backfill can skip the other.
try:
from scribe.services.snippets import backfill_snippet_data
await backfill_snippet_data()
except Exception:
logger.warning("Snippet data backfill failed", exc_info=True)
asyncio.create_task(_delayed_backfill())
+18
View File
@@ -19,9 +19,16 @@ from scribe.services import systems as systems_svc
async def list_snippets(
q: str = "", tag: str = "", limit: int = 50, project_id: int = 0,
repo: str = "", path: str = "", symbol: str = "",
) -> dict:
"""List recorded snippets (reusable functions/components).
Two ways to ask, usable together: by MEANING (`q` — "what do I need this code
to do?") and by PLACE (`repo`/`path`/`symbol` — "what canonical helpers
already live in the file I'm about to edit?"). Reach for the place form
before you write or change code in a file: it is the cheap way to find prior
art you'd otherwise duplicate a few lines down.
Args:
q: Free-text search across name + body (optional). Matches on meaning as
well as wording, so describe what you need the code to DO.
@@ -30,6 +37,16 @@ async def list_snippets(
project_id: Narrow to one project. 0 (default) searches every project —
usually what you want, since a helper you need here may well have
been written somewhere else.
repo: Narrow to snippets recorded in this repo, matched exactly — the
same repo string used when recording, e.g. "Scribe".
path: Narrow to snippets recorded at this path. Matches the exact file
OR anything beneath it, so "frontend/src" finds
"frontend/src/lib/x.ts" as well as itself.
symbol: Narrow to snippets recorded under this symbol name, exactly.
`repo`/`path`/`symbol` must all match the SAME recorded location, so a
snippet that lives in repo A and, separately, at path B in another repo is
not returned for repo=A + path=B.
Returns {"snippets": [{id, title, tags, preview}], "total": int}. The title
reads "name — when to reach for it"; open one in full with get_snippet(id).
@@ -44,6 +61,7 @@ async def list_snippets(
items, total = await snippets_svc.list_snippets(
uid, q=q or None, tag=tag, limit=max(1, min(limit, 100)),
project_id=project_id or None,
repo=repo, path=path, symbol=symbol,
)
return {
"snippets": await access_svc.label_shared_items(uid, items),
+6
View File
@@ -56,9 +56,15 @@ async def list_snippets_route():
project_id = int(request.args.get("project_id", 0) or 0) or None
except (TypeError, ValueError):
project_id = None
# Reverse lookup — "what already lives here?" Empty args are ignored, so the
# plain list is unchanged.
repo = request.args.get("repo", "")
path = request.args.get("path", "")
symbol = request.args.get("symbol", "")
limit, offset = parse_pagination()
items, total = await snippets_svc.list_snippets(
uid, q=q, tag=tag, limit=limit, offset=offset, project_id=project_id,
repo=repo, path=path, symbol=symbol,
)
# Mark rows owned by someone else so the UI can show whose they are — an
# unmarked row in your own list reads as one you recorded and vetted.
+98 -1
View File
@@ -15,6 +15,7 @@ The asymmetry is the point. A record that appears unasked reads as one you
endorsed, so a one-off direct share has to be searched for rather than arriving
in your ambient lists.
"""
import json
import logging
from sqlalchemy import func, select
@@ -28,6 +29,85 @@ logger = logging.getLogger(__name__)
_SNIPPET_LEN = 200
# --- the location filter (reverse lookup, #2083) ------------------------------
#
# ONE predicate in two dialects: "this record carries a location matching every
# part asked for." The SQL form runs in the browse arm and the keyword arm,
# before the count and the page slice, so totals stay honest; the Python form
# runs in the semantic arm, which post-filters candidates it already holds in
# memory. THE TWO MUST CHANGE TOGETHER — a filter only one arm applies makes a
# snippet findable by wording and invisible by location, which is exactly the
# split tests/test_retrieval_scopes.py exists to prevent.
#
# Semantics, both dialects:
# - parts are ANDed WITHIN a single location entry, never across the list. A
# snippet whose first location is in repo A and whose second is at path B
# does not answer repo=A + path=B — it was never in that place.
# - `path` matches exactly OR as a directory prefix: "frontend/src" finds
# "frontend/src/lib/x.ts". Prefix is the one part a GIN containment lookup
# can't serve (migration 0070), which is why this is a jsonpath `@?` rather
# than `@>` — jsonpath `starts with` is index-served by the same GIN index.
#
# The filter reads `notes.data`, so it only sees rows carrying that mirror.
# Rows written before migration 0070 are populated once at startup by
# snippets.backfill_snippet_data — without that, this query would answer "no
# snippets here" for an old snippet and the caller would write the helper again.
LOCATION_KEYS = ("repo", "path", "symbol")
def location_parts(repo: str = "", path: str = "", symbol: str = "") -> dict[str, str]:
"""The non-empty, stripped location parts asked for; {} when none were."""
given = {"repo": repo, "path": path, "symbol": symbol}
return {k: (v or "").strip() for k, v in given.items() if (v or "").strip()}
def _path_matches(have: str, want: str) -> bool:
"""Exact, or `have` sits somewhere under the `want` directory."""
return have == want or have.startswith(want.rstrip("/") + "/")
def location_matches(data: dict | None, parts: dict[str, str]) -> bool:
"""Python dialect of the location predicate. Keep in step with _location_clause."""
if not parts:
return True
for loc in (data or {}).get("locations") or []:
if all(
_path_matches((loc.get(key) or "").strip(), want)
if key == "path"
else (loc.get(key) or "").strip() == want
for key, want in parts.items()
):
return True
return False
def location_jsonpath(parts: dict[str, str]) -> str:
"""The jsonpath behind the SQL dialect — one `locations` entry matching all parts.
Values are embedded as JSON string literals (jsonpath uses JSON quoting), so
a repo or path carrying a quote can't break out of the expression. The keys
are our own fixed set, never caller input.
"""
filters = []
for key in LOCATION_KEYS:
if key not in parts:
continue
want = parts[key]
literal = json.dumps(want)
if key == "path":
prefix = json.dumps(want.rstrip("/") + "/")
filters.append(f"(@.path == {literal} || @.path starts with {prefix})")
else:
filters.append(f"@.{key} == {literal}")
return f"$.locations[*] ? ({' && '.join(filters)})"
def _location_clause(parts: dict[str, str]):
"""SQL dialect of the location predicate. Keep in step with location_matches."""
return Note.data.path_exists(location_jsonpath(parts))
def _note_to_item(note: Note) -> dict:
item: dict = {
"id": note.id,
@@ -80,18 +160,24 @@ async def query_knowledge(
limit: int,
offset: int,
project_id: int | None = None,
locations: dict[str, str] | None = None,
) -> tuple[list[dict], int]:
"""Query knowledge objects (non-task notes) with filters.
`project_id` narrows to one project (None = every project).
`locations` narrows to records whose `data.locations` holds an entry matching
every part given — build it with `location_parts(repo=…, path=…, symbol=…)`.
Today only snippets carry locations, but the column is general, so the filter
lives here with the query rather than in one type's service.
Returns (items, total_count).
"""
# Semantic search path — scores take priority over sort
if q:
return await _semantic_knowledge_search(
user_id, q, note_type=note_type, tags=tags, limit=limit,
offset=offset, project_id=project_id,
offset=offset, project_id=project_id, locations=locations,
)
# No query = browsing. Narrower scope: a record shared directly with the
@@ -107,6 +193,9 @@ async def query_knowledge(
for tag in tags:
base = base.where(Note.tags.contains([tag]))
if locations:
base = base.where(_location_clause(locations))
# Count before pagination
count_stmt = select(func.count()).select_from(base.subquery())
total: int = (await session.execute(count_stmt)).scalar_one()
@@ -134,6 +223,7 @@ async def _semantic_knowledge_search(
limit: int,
offset: int,
project_id: int | None = None,
locations: dict[str, str] | None = None,
) -> tuple[list[dict], int]:
"""Hybrid search: keyword matches first (title/body ILIKE), then semantic results.
@@ -167,6 +257,8 @@ async def _semantic_knowledge_search(
base = base.where(Note.project_id == project_id)
for tag in tags:
base = base.where(Note.tags.contains([tag]))
if locations:
base = base.where(_location_clause(locations))
# Title matches first, then body-only matches, newest first within each
base = base.order_by(
Note.title.ilike(pattern).desc(),
@@ -204,6 +296,11 @@ async def _semantic_knowledge_search(
continue
if tags and not all(t in (note.tags or []) for t in tags):
continue
# The Python dialect of the same predicate the SQL arms apply above —
# these candidates arrive already fetched, so there's no query to
# narrow. See the comment on location_matches.
if locations and not location_matches(note.data, locations):
continue
semantic_notes.append(note)
except Exception:
logger.warning("Semantic search unavailable, using keyword results only", exc_info=True)
+86 -7
View File
@@ -5,10 +5,10 @@ component recorded once so a later session can recall it before writing a
one-off. It carries a name, language, signature, canonical location
(repo · path · symbol), a one-line "when to reach for it", and the code itself.
Structured fields are stored as a **body-convention** — no dedicated column, so
a snippet inherits everything a note has (embeddings, ACL, project/System
association, dedup) and, crucially, becomes eligible for semantic recall the
moment it's embedded:
Structured fields are stored as a **body-convention** — the body is the readable
form and the thing that gets embedded, so a snippet inherits everything a note
has (embeddings, ACL, project/System association, dedup) and, crucially, becomes
eligible for semantic recall the moment it's embedded:
- ``title`` = ``"{name}{when_to_use}"``. The title is exactly what the
title-first auto-inject surfaces, so this one line self-describes the snippet
@@ -17,18 +17,30 @@ moment it's embedded:
- ``body`` = templated markdown (When to use / Signature / Location, then a
fenced code block).
Since migration 0070 the same fields are ALSO written to ``notes.data`` (see
``compose_data``) — a queryable mirror, not a second source of truth: it carries
no code, and it exists so location/language can be indexed rather than regexed
out of every body. Reads prefer it and fall back to the body parse.
The public field API (name/language/signature/location/when_to_use/code) lives
here, so the underlying storage could later move to a structured column without
changing any caller (MCP tool, REST route, UI).
here, so callers (MCP tool, REST route, UI) never see which of the two a field
came from.
"""
from __future__ import annotations
import asyncio
import logging
import re
from sqlalchemy import select
from scribe.models import async_session
from scribe.models.note import Note
from scribe.services import knowledge as knowledge_svc
from scribe.services import notes as notes_svc
logger = logging.getLogger(__name__)
SNIPPET_NOTE_TYPE = "snippet"
SNIPPET_TAG = "snippet"
@@ -354,6 +366,63 @@ def snippet_fields(note) -> dict:
return merged
async def backfill_snippet_data(*, batch: int = 500) -> int:
"""Populate `notes.data` for snippet rows that predate migration 0070.
Runs once at startup (see app.py). Returns how many rows were filled.
Migration 0070 deliberately left `data` NULL on existing rows, because
*reading* a snippet never needed it — `snippet_fields` falls back to parsing
the body. Querying does: the location reverse lookup (#2083) is a jsonpath
over this column, so a NULL-`data` snippet would be reported as "nothing
recorded here" and the caller would write the helper again — silently worse
than having no reverse lookup at all. Every consumer of the column
(reverse lookup, and the write-path trigger built on it) can then assume the
mirror is present instead of carrying a second body-regex arm.
This does not reverse 0070's actual caution, which was about mangling a
hand-edited *body*: the body is never touched here, only the mirror derived
from it, by the same parser the read path already trusts. Idempotent — a
filled row is skipped forever after, and a snippet with no structured fields
at all settles at `{}` rather than staying NULL and being re-scanned. Trashed
rows are included so a later restore comes back queryable.
"""
filled = 0
async with async_session() as session:
while True:
rows = list(
(
await session.execute(
select(Note)
.where(Note.note_type == SNIPPET_NOTE_TYPE)
.where(Note.data.is_(None))
.limit(batch)
)
)
.scalars()
.all()
)
if not rows:
break
for note in rows:
fields = parse_snippet_fields(note.title, note.body, note.tags)
note.data = compose_data(
name=fields["name"],
when_to_use=fields["when_to_use"],
signature=fields["signature"],
language=fields["language"],
locations=fields["locations"],
merged_from=fields["merged_from"],
)
await session.commit()
filled += len(rows)
if len(rows) < batch:
break
if filled:
logger.info("Snippet data backfill: populated `data` for %d snippet(s)", filled)
return filled
def snippet_to_dict(note) -> dict:
"""Note serialization plus a parsed ``snippet`` sub-object of structured
fields, so callers get both the raw record and the typed view.
@@ -433,12 +502,20 @@ async def list_snippets(
limit: int = 50,
offset: int = 0,
project_id: int | None = None,
repo: str = "",
path: str = "",
symbol: str = "",
) -> tuple[list[dict], int]:
"""List snippets (id/title/tags/preview dicts), most-recently-updated first.
``project_id`` narrows to one project; omit it to reach across every project
— which is the point when the thing you're about to write was already solved
somewhere else."""
somewhere else.
``repo`` / ``path`` / ``symbol`` are the reverse lookup: "what canonical
helpers already live here?" They narrow to snippets recorded at a matching
location, ANDed within one location entry, with ``path`` also matching as a
directory prefix. Combinable with ``q`` — search *and* place."""
return await knowledge_svc.query_knowledge(
user_id=user_id,
note_type=SNIPPET_NOTE_TYPE,
@@ -448,6 +525,8 @@ async def list_snippets(
limit=max(1, min(limit, 100)),
offset=max(0, offset),
project_id=project_id,
locations=knowledge_svc.location_parts(repo=repo, path=path, symbol=symbol)
or None,
)