feat(snippets): reverse lookup — find snippets by repo/path/symbol
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Failing after 27s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 1m42s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Failing after 27s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 1m42s
"What canonical helpers already live in this file?" was unanswerable: location lived only in the body markdown. It is now a jsonpath containment query over the `notes.data` mirror added by migration 0070. - One predicate in two dialects in services/knowledge.py: SQL (`data @?`, applied in the browse arm and the keyword arm before count/pagination, so totals stay honest) and Python (`location_matches`, for the semantic arm which post-filters candidates it already holds). Both must change together. - Parts are ANDed within a SINGLE locations entry — repo A in one entry and path B in another is not "recorded at A/B". `path` also matches as a directory prefix, via jsonpath `starts with` rather than `@>`, which the same GIN index serves. - `repo`/`path`/`symbol` reach the service, the REST list and the MCP tool under one name with one default (rule #33); the MCP docstring teaches the place form, and so does the reusing-code skill (plugin.json bumped). - UI: a Location disclosure beside the snippet search, with its own empty state — "nothing kept there, so what you're about to write is new." Settles #2083's open question (pre-0070 NULL `data`) by backfilling after all: `backfill_snippet_data` runs at startup, deriving the mirror from the body with the same parser the read path trusts. 0070's caution was about mangling a hand-edited body; this never touches the body. The alternative was a permanent second body-regex arm, or a query that silently answers "nothing here" for an old snippet and gets the helper written twice. Refs #2083, milestone #232.
This commit is contained in:
@@ -5,10 +5,10 @@ component recorded once so a later session can recall it before writing a
|
||||
one-off. It carries a name, language, signature, canonical location
|
||||
(repo · path · symbol), a one-line "when to reach for it", and the code itself.
|
||||
|
||||
Structured fields are stored as a **body-convention** — no dedicated column, so
|
||||
a snippet inherits everything a note has (embeddings, ACL, project/System
|
||||
association, dedup) and, crucially, becomes eligible for semantic recall the
|
||||
moment it's embedded:
|
||||
Structured fields are stored as a **body-convention** — the body is the readable
|
||||
form and the thing that gets embedded, so a snippet inherits everything a note
|
||||
has (embeddings, ACL, project/System association, dedup) and, crucially, becomes
|
||||
eligible for semantic recall the moment it's embedded:
|
||||
|
||||
- ``title`` = ``"{name} — {when_to_use}"``. The title is exactly what the
|
||||
title-first auto-inject surfaces, so this one line self-describes the snippet
|
||||
@@ -17,18 +17,30 @@ moment it's embedded:
|
||||
- ``body`` = templated markdown (When to use / Signature / Location, then a
|
||||
fenced code block).
|
||||
|
||||
Since migration 0070 the same fields are ALSO written to ``notes.data`` (see
|
||||
``compose_data``) — a queryable mirror, not a second source of truth: it carries
|
||||
no code, and it exists so location/language can be indexed rather than regexed
|
||||
out of every body. Reads prefer it and fall back to the body parse.
|
||||
|
||||
The public field API (name/language/signature/location/when_to_use/code) lives
|
||||
here, so the underlying storage could later move to a structured column without
|
||||
changing any caller (MCP tool, REST route, UI).
|
||||
here, so callers (MCP tool, REST route, UI) never see which of the two a field
|
||||
came from.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.note import Note
|
||||
from scribe.services import knowledge as knowledge_svc
|
||||
from scribe.services import notes as notes_svc
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SNIPPET_NOTE_TYPE = "snippet"
|
||||
SNIPPET_TAG = "snippet"
|
||||
|
||||
@@ -354,6 +366,63 @@ def snippet_fields(note) -> dict:
|
||||
return merged
|
||||
|
||||
|
||||
async def backfill_snippet_data(*, batch: int = 500) -> int:
|
||||
"""Populate `notes.data` for snippet rows that predate migration 0070.
|
||||
|
||||
Runs once at startup (see app.py). Returns how many rows were filled.
|
||||
|
||||
Migration 0070 deliberately left `data` NULL on existing rows, because
|
||||
*reading* a snippet never needed it — `snippet_fields` falls back to parsing
|
||||
the body. Querying does: the location reverse lookup (#2083) is a jsonpath
|
||||
over this column, so a NULL-`data` snippet would be reported as "nothing
|
||||
recorded here" and the caller would write the helper again — silently worse
|
||||
than having no reverse lookup at all. Every consumer of the column
|
||||
(reverse lookup, and the write-path trigger built on it) can then assume the
|
||||
mirror is present instead of carrying a second body-regex arm.
|
||||
|
||||
This does not reverse 0070's actual caution, which was about mangling a
|
||||
hand-edited *body*: the body is never touched here, only the mirror derived
|
||||
from it, by the same parser the read path already trusts. Idempotent — a
|
||||
filled row is skipped forever after, and a snippet with no structured fields
|
||||
at all settles at `{}` rather than staying NULL and being re-scanned. Trashed
|
||||
rows are included so a later restore comes back queryable.
|
||||
"""
|
||||
filled = 0
|
||||
async with async_session() as session:
|
||||
while True:
|
||||
rows = list(
|
||||
(
|
||||
await session.execute(
|
||||
select(Note)
|
||||
.where(Note.note_type == SNIPPET_NOTE_TYPE)
|
||||
.where(Note.data.is_(None))
|
||||
.limit(batch)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if not rows:
|
||||
break
|
||||
for note in rows:
|
||||
fields = parse_snippet_fields(note.title, note.body, note.tags)
|
||||
note.data = compose_data(
|
||||
name=fields["name"],
|
||||
when_to_use=fields["when_to_use"],
|
||||
signature=fields["signature"],
|
||||
language=fields["language"],
|
||||
locations=fields["locations"],
|
||||
merged_from=fields["merged_from"],
|
||||
)
|
||||
await session.commit()
|
||||
filled += len(rows)
|
||||
if len(rows) < batch:
|
||||
break
|
||||
if filled:
|
||||
logger.info("Snippet data backfill: populated `data` for %d snippet(s)", filled)
|
||||
return filled
|
||||
|
||||
|
||||
def snippet_to_dict(note) -> dict:
|
||||
"""Note serialization plus a parsed ``snippet`` sub-object of structured
|
||||
fields, so callers get both the raw record and the typed view.
|
||||
@@ -433,12 +502,20 @@ async def list_snippets(
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
project_id: int | None = None,
|
||||
repo: str = "",
|
||||
path: str = "",
|
||||
symbol: str = "",
|
||||
) -> tuple[list[dict], int]:
|
||||
"""List snippets (id/title/tags/preview dicts), most-recently-updated first.
|
||||
|
||||
``project_id`` narrows to one project; omit it to reach across every project
|
||||
— which is the point when the thing you're about to write was already solved
|
||||
somewhere else."""
|
||||
somewhere else.
|
||||
|
||||
``repo`` / ``path`` / ``symbol`` are the reverse lookup: "what canonical
|
||||
helpers already live here?" They narrow to snippets recorded at a matching
|
||||
location, ANDed within one location entry, with ``path`` also matching as a
|
||||
directory prefix. Combinable with ``q`` — search *and* place."""
|
||||
return await knowledge_svc.query_knowledge(
|
||||
user_id=user_id,
|
||||
note_type=SNIPPET_NOTE_TYPE,
|
||||
@@ -448,6 +525,8 @@ async def list_snippets(
|
||||
limit=max(1, min(limit, 100)),
|
||||
offset=max(0, offset),
|
||||
project_id=project_id,
|
||||
locations=knowledge_svc.location_parts(repo=repo, path=path, symbol=symbol)
|
||||
or None,
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user