feat(snippets): reverse lookup — find snippets by repo/path/symbol
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Failing after 27s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 1m42s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Failing after 27s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 1m42s
"What canonical helpers already live in this file?" was unanswerable: location lived only in the body markdown. It is now a jsonpath containment query over the `notes.data` mirror added by migration 0070. - One predicate in two dialects in services/knowledge.py: SQL (`data @?`, applied in the browse arm and the keyword arm before count/pagination, so totals stay honest) and Python (`location_matches`, for the semantic arm which post-filters candidates it already holds). Both must change together. - Parts are ANDed within a SINGLE locations entry — repo A in one entry and path B in another is not "recorded at A/B". `path` also matches as a directory prefix, via jsonpath `starts with` rather than `@>`, which the same GIN index serves. - `repo`/`path`/`symbol` reach the service, the REST list and the MCP tool under one name with one default (rule #33); the MCP docstring teaches the place form, and so does the reusing-code skill (plugin.json bumped). - UI: a Location disclosure beside the snippet search, with its own empty state — "nothing kept there, so what you're about to write is new." Settles #2083's open question (pre-0070 NULL `data`) by backfilling after all: `backfill_snippet_data` runs at startup, deriving the mirror from the body with the same parser the read path trusts. 0070's caution was about mangling a hand-edited body; this never touches the body. The alternative was a permanent second body-regex arm, or a query that silently answers "nothing here" for an old snippet and gets the helper written twice. Refs #2083, milestone #232.
This commit is contained in:
@@ -15,6 +15,7 @@ The asymmetry is the point. A record that appears unasked reads as one you
|
||||
endorsed, so a one-off direct share has to be searched for rather than arriving
|
||||
in your ambient lists.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
|
||||
from sqlalchemy import func, select
|
||||
@@ -28,6 +29,85 @@ logger = logging.getLogger(__name__)
|
||||
_SNIPPET_LEN = 200
|
||||
|
||||
|
||||
# --- the location filter (reverse lookup, #2083) ------------------------------
|
||||
#
|
||||
# ONE predicate in two dialects: "this record carries a location matching every
|
||||
# part asked for." The SQL form runs in the browse arm and the keyword arm,
|
||||
# before the count and the page slice, so totals stay honest; the Python form
|
||||
# runs in the semantic arm, which post-filters candidates it already holds in
|
||||
# memory. THE TWO MUST CHANGE TOGETHER — a filter only one arm applies makes a
|
||||
# snippet findable by wording and invisible by location, which is exactly the
|
||||
# split tests/test_retrieval_scopes.py exists to prevent.
|
||||
#
|
||||
# Semantics, both dialects:
|
||||
# - parts are ANDed WITHIN a single location entry, never across the list. A
|
||||
# snippet whose first location is in repo A and whose second is at path B
|
||||
# does not answer repo=A + path=B — it was never in that place.
|
||||
# - `path` matches exactly OR as a directory prefix: "frontend/src" finds
|
||||
# "frontend/src/lib/x.ts". Prefix is the one part a GIN containment lookup
|
||||
# can't serve (migration 0070), which is why this is a jsonpath `@?` rather
|
||||
# than `@>` — jsonpath `starts with` is index-served by the same GIN index.
|
||||
#
|
||||
# The filter reads `notes.data`, so it only sees rows carrying that mirror.
|
||||
# Rows written before migration 0070 are populated once at startup by
|
||||
# snippets.backfill_snippet_data — without that, this query would answer "no
|
||||
# snippets here" for an old snippet and the caller would write the helper again.
|
||||
|
||||
LOCATION_KEYS = ("repo", "path", "symbol")
|
||||
|
||||
|
||||
def location_parts(repo: str = "", path: str = "", symbol: str = "") -> dict[str, str]:
|
||||
"""The non-empty, stripped location parts asked for; {} when none were."""
|
||||
given = {"repo": repo, "path": path, "symbol": symbol}
|
||||
return {k: (v or "").strip() for k, v in given.items() if (v or "").strip()}
|
||||
|
||||
|
||||
def _path_matches(have: str, want: str) -> bool:
|
||||
"""Exact, or `have` sits somewhere under the `want` directory."""
|
||||
return have == want or have.startswith(want.rstrip("/") + "/")
|
||||
|
||||
|
||||
def location_matches(data: dict | None, parts: dict[str, str]) -> bool:
|
||||
"""Python dialect of the location predicate. Keep in step with _location_clause."""
|
||||
if not parts:
|
||||
return True
|
||||
for loc in (data or {}).get("locations") or []:
|
||||
if all(
|
||||
_path_matches((loc.get(key) or "").strip(), want)
|
||||
if key == "path"
|
||||
else (loc.get(key) or "").strip() == want
|
||||
for key, want in parts.items()
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def location_jsonpath(parts: dict[str, str]) -> str:
|
||||
"""The jsonpath behind the SQL dialect — one `locations` entry matching all parts.
|
||||
|
||||
Values are embedded as JSON string literals (jsonpath uses JSON quoting), so
|
||||
a repo or path carrying a quote can't break out of the expression. The keys
|
||||
are our own fixed set, never caller input.
|
||||
"""
|
||||
filters = []
|
||||
for key in LOCATION_KEYS:
|
||||
if key not in parts:
|
||||
continue
|
||||
want = parts[key]
|
||||
literal = json.dumps(want)
|
||||
if key == "path":
|
||||
prefix = json.dumps(want.rstrip("/") + "/")
|
||||
filters.append(f"(@.path == {literal} || @.path starts with {prefix})")
|
||||
else:
|
||||
filters.append(f"@.{key} == {literal}")
|
||||
return f"$.locations[*] ? ({' && '.join(filters)})"
|
||||
|
||||
|
||||
def _location_clause(parts: dict[str, str]):
|
||||
"""SQL dialect of the location predicate. Keep in step with location_matches."""
|
||||
return Note.data.path_exists(location_jsonpath(parts))
|
||||
|
||||
|
||||
def _note_to_item(note: Note) -> dict:
|
||||
item: dict = {
|
||||
"id": note.id,
|
||||
@@ -80,18 +160,24 @@ async def query_knowledge(
|
||||
limit: int,
|
||||
offset: int,
|
||||
project_id: int | None = None,
|
||||
locations: dict[str, str] | None = None,
|
||||
) -> tuple[list[dict], int]:
|
||||
"""Query knowledge objects (non-task notes) with filters.
|
||||
|
||||
`project_id` narrows to one project (None = every project).
|
||||
|
||||
`locations` narrows to records whose `data.locations` holds an entry matching
|
||||
every part given — build it with `location_parts(repo=…, path=…, symbol=…)`.
|
||||
Today only snippets carry locations, but the column is general, so the filter
|
||||
lives here with the query rather than in one type's service.
|
||||
|
||||
Returns (items, total_count).
|
||||
"""
|
||||
# Semantic search path — scores take priority over sort
|
||||
if q:
|
||||
return await _semantic_knowledge_search(
|
||||
user_id, q, note_type=note_type, tags=tags, limit=limit,
|
||||
offset=offset, project_id=project_id,
|
||||
offset=offset, project_id=project_id, locations=locations,
|
||||
)
|
||||
|
||||
# No query = browsing. Narrower scope: a record shared directly with the
|
||||
@@ -107,6 +193,9 @@ async def query_knowledge(
|
||||
for tag in tags:
|
||||
base = base.where(Note.tags.contains([tag]))
|
||||
|
||||
if locations:
|
||||
base = base.where(_location_clause(locations))
|
||||
|
||||
# Count before pagination
|
||||
count_stmt = select(func.count()).select_from(base.subquery())
|
||||
total: int = (await session.execute(count_stmt)).scalar_one()
|
||||
@@ -134,6 +223,7 @@ async def _semantic_knowledge_search(
|
||||
limit: int,
|
||||
offset: int,
|
||||
project_id: int | None = None,
|
||||
locations: dict[str, str] | None = None,
|
||||
) -> tuple[list[dict], int]:
|
||||
"""Hybrid search: keyword matches first (title/body ILIKE), then semantic results.
|
||||
|
||||
@@ -167,6 +257,8 @@ async def _semantic_knowledge_search(
|
||||
base = base.where(Note.project_id == project_id)
|
||||
for tag in tags:
|
||||
base = base.where(Note.tags.contains([tag]))
|
||||
if locations:
|
||||
base = base.where(_location_clause(locations))
|
||||
# Title matches first, then body-only matches, newest first within each
|
||||
base = base.order_by(
|
||||
Note.title.ilike(pattern).desc(),
|
||||
@@ -204,6 +296,11 @@ async def _semantic_knowledge_search(
|
||||
continue
|
||||
if tags and not all(t in (note.tags or []) for t in tags):
|
||||
continue
|
||||
# The Python dialect of the same predicate the SQL arms apply above —
|
||||
# these candidates arrive already fetched, so there's no query to
|
||||
# narrow. See the comment on location_matches.
|
||||
if locations and not location_matches(note.data, locations):
|
||||
continue
|
||||
semantic_notes.append(note)
|
||||
except Exception:
|
||||
logger.warning("Semantic search unavailable, using keyword results only", exc_info=True)
|
||||
|
||||
Reference in New Issue
Block a user