fix(knowledge): the browse vocabulary catches up three kinds, and a snippet's mirror survives the generic door (#3128 recs 2-6)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / integration (push) Successful in 27s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 33s

Spike #3128 found the storage sound and the retrieval vocabulary frozen
before `issue` shipped (0065). Five things, in the order they had to land.

**The mirror (rec 5, the data-integrity one).** `notes.data` is DERIVED from
a snippet's body, but only `update_snippet` knew that. `update_note` is a
hasattr loop with no snippet awareness, and both doors reach it — so PATCH
/api/notes/<snippet_id> {body} rewrote the body and left the mirror behind.
`snippet_fields` PREFERS the mirror, so the row went on reporting its old
repo/path/symbol to the location reverse lookup and to prior-art recall while
displaying its new body: surfaced with full authority, and wrong.
`snippets.recompose_data` rebuilds it from the body, carrying `verification`
and `provenance` (neither is in the body to parse). An explicit `data` still
wins, so every snippet-service write is untouched.

**One facet table (rec 3), before adding any facet.** The type predicate was
written three times — SQL, Python over semantic candidates, and a ternary
computing the `is_task` pre-filter — and agreed only by luck. Adding `issue`
to the SQL arm alone would have set the pre-filter to is_task=False, handed
the Python arm a candidate set with no tasks in it, and returned an empty
semantic half for the Issues facet forever with nothing red. `_FACETS` now
generates all three. The Python arm also regains the `status IS NULL` half its
SQL twin always had.

**Issue and spike become facets (rec 2).** 435 issues — 17% of every task —
were filterable nowhere on the human surface, while retired `plan` (90 rows)
had a chip of its own. `_VALID_TYPES` was a hand-kept copy and is now derived.
`plan` stays a valid facet for its legacy rows; it loses its chip.

**Snippets stop being half-present in the feed (rec 4).** All 90 were in the
All list, in no count, wearing an empty badge, and opening in the note editor.
Counts now group by task_kind — every kind for the same two round-trips, which
is why `issue` had no number — and total includes snippets, so the All chip
matches the list it labels. Snippet cards route to /snippets/:id.

**The prose that excused it (rec 6).** `snippet_fields` and the `data` column
both still said pre-0070 rows were "never backfilled". True when 0070 landed,
false since `backfill_snippet_data` shipped, and it read as licence for a
stale mirror.

Tests: the pre-filter can never exclude a row its own facet accepts (the
regression, parameterised over every facet); both dialects select exactly
their own rows; an unknown facet matches nothing; the mirror follows a body or
title write, carries the verdict, and yields to an explicit `data`.
`compiled_sql` moves to tests/helpers rather than becoming a third copy.

Write-up: note #3161.
This commit is contained in:
2026-08-28 12:06:43 -04:00
parent d0a2733cb6
commit f80401d58e
11 changed files with 576 additions and 104 deletions
+116 -59
View File
@@ -1,4 +1,4 @@
"""Knowledge service — unified query across notes, tasks, plans, and processes.
"""Knowledge service — one query across every record kind Scribe holds.
ACL (rules #47/#78, decision note 2094): these queries were owner-only until
2026-07-25, which meant a record shared with you could be opened by id but never
@@ -265,22 +265,94 @@ def _note_to_item(note: Note) -> dict:
return item
def _apply_type_filter(stmt, note_type: str | None):
"""Apply the type facet to a Note select.
# What each type facet MEANS, once, for every arm that has to know.
#
# The vocabulary spans BOTH typing axes — `note_type` for non-task records and
# `task_kind` for tasks — so a facet cannot be a filter on one column, which is
# why this is a table rather than a chain of ifs. Each entry is
# (is_task, the value pinned on that axis); None pins nothing, i.e. every task.
#
# It is a table because the alternative had already gone wrong. The predicate
# was written three times — a SQL if-chain, a Python if-chain over semantic
# candidates, and a ternary computing the `is_task` pre-filter — and the three
# only agreed by luck. Adding `issue` to the SQL arm alone (the obvious edit,
# and the one #3128 was about to make) would have set the pre-filter to
# is_task=False, handed the Python arm a candidate set containing no tasks at
# all, and returned an empty semantic half for the Issues facet forever, with
# nothing red anywhere. A new facet is now one row here.
#
# `plan` is retired (0066) but kept: 90 legacy plan-tasks exist and a facet
# they answer to costs one line. It simply has no chip in the UI any more.
_FACETS: dict[str, tuple[bool, str | None]] = {
"task": (True, None),
"work": (True, "work"),
"issue": (True, "issue"),
"spike": (True, "spike"),
"plan": (True, "plan"),
"note": (False, "note"),
"process": (False, "process"),
"snippet": (False, "snippet"),
}
'task' = any task (status not null); 'plan' = a task with task_kind='plan';
any other non-empty type = a non-task note of that note_type; None = all.
# The non-task record types, for the counts query. Derived so it cannot drift
# from the table above.
NON_TASK_FACETS = tuple(
value for _is_task, value in _FACETS.values() if not _is_task and value
)
Trashed rows (deleted_at set) are always excluded.
# The whole vocabulary, for the door's request validation — public so the route
# validates against the same table the query reads instead of a hand-kept copy.
FACET_TYPES = frozenset(_FACETS)
# An unrecognised facet resolves to "a non-task note whose note_type is that
# string" — which matches nothing, since no row stores an unknown type. That is
# the behaviour the old if-chain had by falling through, and it is the right
# one: a typo should return an empty list, never the whole corpus.
def _facet(note_type: str) -> tuple[bool, str | None]:
return _FACETS.get(note_type, (False, note_type))
def facet_is_task(note_type: str | None) -> bool | None:
"""The `is_task` pre-filter a facet implies — None when it spans both.
Used to narrow the semantic candidate set before it is fetched. Reads the
same table `_apply_type_filter` and `matches_facet` read, so the pre-filter
can no longer disagree with the predicate it is meant to anticipate.
"""
if not note_type:
return None
return _facet(note_type)[0]
def matches_facet(note, note_type: str | None) -> bool:
"""The Python dialect of `_apply_type_filter`, for candidates the vector
search has already fetched — there is no query left to narrow.
Generated from the same table, so this is a translation rather than a
second implementation. Note the `not note.is_task` arm: the hand-written
version omitted it and was saved only by the upstream pre-filter.
"""
if not note_type:
return True
is_task, value = _facet(note_type)
if is_task:
return note.is_task and (value is None or note.task_kind == value)
return not note.is_task and note.note_type == value
def _apply_type_filter(stmt, note_type: str | None):
"""Apply the type facet to a Note select. Trashed rows are always excluded."""
stmt = stmt.where(Note.deleted_at.is_(None))
if note_type == "task":
return stmt.where(Note.status.isnot(None))
if note_type == "plan":
return stmt.where(Note.status.isnot(None)).where(Note.task_kind == "plan")
if note_type:
return stmt.where(Note.note_type == note_type).where(Note.status.is_(None))
return stmt
if not note_type:
return stmt
is_task, value = _facet(note_type)
if is_task:
stmt = stmt.where(Note.status.isnot(None))
if value is not None:
stmt = stmt.where(Note.task_kind == value)
return stmt
return stmt.where(Note.status.is_(None)).where(Note.note_type == value)
async def query_knowledge(
@@ -295,7 +367,7 @@ async def query_knowledge(
locations: dict[str, str] | None = None,
verification: str = "",
) -> tuple[list[dict], int]:
"""Query knowledge objects (non-task notes) with filters.
"""Query knowledge objects with filters.
`project_id` narrows to one project (None = every project).
@@ -424,7 +496,7 @@ async def _semantic_knowledge_search(
INTERACTIVE_SEARCH_THRESHOLD,
semantic_search_notes,
)
is_task_filter = True if note_type in ("task", "plan") else (False if note_type else None)
is_task_filter = facet_is_task(note_type)
import time as _time
_t0 = _time.perf_counter()
candidates = await semantic_search_notes(
@@ -454,11 +526,7 @@ async def _semantic_knowledge_search(
for _score, note in candidates:
if note.deleted_at is not None:
continue
if note_type == "task" and not note.is_task:
continue
elif note_type == "plan" and (not note.is_task or note.task_kind != "plan"):
continue
elif note_type and note_type not in ("task", "plan") and note.note_type != note_type:
if not matches_facet(note, note_type):
continue
if tags and not all(t in (note.tags or []) for t in tags):
continue
@@ -514,51 +582,40 @@ async def get_knowledge_counts(user_id: int, tags: list[str] | None = None) -> d
search would surface."""
visible = browsable_notes_clause(user_id)
async with async_session() as session:
# Count non-task types
stmt = (
select(Note.note_type, func.count(Note.id))
.where(visible)
.where(Note.status.is_(None))
.where(Note.deleted_at.is_(None))
.where(Note.note_type.in_(["note", "process"]))
.group_by(Note.note_type)
)
if tags:
for tag in tags:
def _scoped(stmt):
stmt = stmt.where(visible).where(Note.deleted_at.is_(None))
for tag in tags or []:
stmt = stmt.where(Note.tags.contains([tag]))
rows = list((await session.execute(stmt)).all())
counts = {row[0]: row[1] for row in rows}
return stmt
# Count tasks separately (is_task = status IS NOT NULL)
task_stmt = (
select(func.count(Note.id))
.where(visible)
# One grouped query per typing axis. The task half used to be a count
# for 'task' plus a second count for 'plan', which is why 'issue' —
# 17% of every task here — had no number to show: each kind needed its
# own query and nobody added one. Grouping by task_kind counts every
# kind, including ones added later, for the same two round-trips.
non_task = _scoped(
select(Note.note_type, func.count(Note.id))
.where(Note.status.is_(None))
.where(Note.note_type.in_(NON_TASK_FACETS))
).group_by(Note.note_type)
counts = {t: n for t, n in (await session.execute(non_task)).all()}
by_kind = _scoped(
select(Note.task_kind, func.count(Note.id))
.where(Note.status.isnot(None))
.where(Note.deleted_at.is_(None))
)
if tags:
for tag in tags:
task_stmt = task_stmt.where(Note.tags.contains([tag]))
task_count: int = (await session.execute(task_stmt)).scalar_one()
counts["task"] = task_count
).group_by(Note.task_kind)
kind_counts = {k: n for k, n in (await session.execute(by_kind)).all()}
# Plans are a subset of tasks (task_kind='plan'); counted for the facet
# but NOT added to total to avoid double-counting against "task".
plan_stmt = (
select(func.count(Note.id))
.where(visible)
.where(Note.status.isnot(None))
.where(Note.task_kind == "plan")
.where(Note.deleted_at.is_(None))
)
if tags:
for tag in tags:
plan_stmt = plan_stmt.where(Note.tags.contains([tag]))
counts["plan"] = (await session.execute(plan_stmt)).scalar_one()
# Kinds are SUBSETS of 'task' and are deliberately left out of the total —
# adding them would count every task twice.
counts["task"] = sum(kind_counts.values())
for kind, value in _FACETS.items():
if value[0] and value[1] is not None:
counts[kind] = kind_counts.get(kind, 0)
for t in ("note", "task", "plan", "process"):
for t in NON_TASK_FACETS:
counts.setdefault(t, 0)
counts["total"] = sum(counts[t] for t in ("note", "task", "process"))
counts["total"] = counts["task"] + sum(counts[t] for t in NON_TASK_FACETS)
return counts