fix(knowledge): the browse vocabulary catches up three kinds, and a snippet's mirror survives the generic door (#3128 recs 2-6)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / integration (push) Successful in 27s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 33s

Spike #3128 found the storage sound and the retrieval vocabulary frozen
before `issue` shipped (0065). Five things, in the order they had to land.

**The mirror (rec 5, the data-integrity one).** `notes.data` is DERIVED from
a snippet's body, but only `update_snippet` knew that. `update_note` is a
hasattr loop with no snippet awareness, and both doors reach it — so PATCH
/api/notes/<snippet_id> {body} rewrote the body and left the mirror behind.
`snippet_fields` PREFERS the mirror, so the row went on reporting its old
repo/path/symbol to the location reverse lookup and to prior-art recall while
displaying its new body: surfaced with full authority, and wrong.
`snippets.recompose_data` rebuilds it from the body, carrying `verification`
and `provenance` (neither is in the body to parse). An explicit `data` still
wins, so every snippet-service write is untouched.

**One facet table (rec 3), before adding any facet.** The type predicate was
written three times — SQL, Python over semantic candidates, and a ternary
computing the `is_task` pre-filter — and agreed only by luck. Adding `issue`
to the SQL arm alone would have set the pre-filter to is_task=False, handed
the Python arm a candidate set with no tasks in it, and returned an empty
semantic half for the Issues facet forever with nothing red. `_FACETS` now
generates all three. The Python arm also regains the `status IS NULL` half its
SQL twin always had.

**Issue and spike become facets (rec 2).** 435 issues — 17% of every task —
were filterable nowhere on the human surface, while retired `plan` (90 rows)
had a chip of its own. `_VALID_TYPES` was a hand-kept copy and is now derived.
`plan` stays a valid facet for its legacy rows; it loses its chip.

**Snippets stop being half-present in the feed (rec 4).** All 90 were in the
All list, in no count, wearing an empty badge, and opening in the note editor.
Counts now group by task_kind — every kind for the same two round-trips, which
is why `issue` had no number — and total includes snippets, so the All chip
matches the list it labels. Snippet cards route to /snippets/:id.

**The prose that excused it (rec 6).** `snippet_fields` and the `data` column
both still said pre-0070 rows were "never backfilled". True when 0070 landed,
false since `backfill_snippet_data` shipped, and it read as licence for a
stale mirror.

Tests: the pre-filter can never exclude a row its own facet accepts (the
regression, parameterised over every facet); both dialects select exactly
their own rows; an unknown facet matches nothing; the mirror follows a body or
title write, carries the verdict, and yields to an explicit `data`.
`compiled_sql` moves to tests/helpers rather than becoming a third copy.

Write-up: note #3161.
This commit is contained in:
2026-08-28 12:06:43 -04:00
parent d0a2733cb6
commit f80401d58e
11 changed files with 576 additions and 104 deletions
+57 -10
View File
@@ -24,7 +24,7 @@ const router = useRouter();
interface KnowledgeItem { interface KnowledgeItem {
id: number; id: number;
note_type: "note" | "task" | "process"; note_type: "note" | "task" | "process" | "snippet";
title: string; title: string;
snippet: string; snippet: string;
tags: string[]; tags: string[];
@@ -42,9 +42,34 @@ interface KnowledgeItem {
task_kind?: TaskKind; task_kind?: TaskKind;
} }
// ─── The facet vocabulary ─────────────────────────────────────────────────────
// Mirrors services/knowledge._FACETS, which is where it is defined for real.
// A facet spans BOTH typing axes — a record TYPE (note / process / snippet) or
// a task KIND (`task` for any, else issue / spike) — because that is what this
// feed actually holds.
//
// `plan` is still a valid facet at the API, for the 90 legacy plan-tasks, but
// it has no chip: retired in 0066, it kept a chip of its own for longer than
// `issue` — 17% of every task here — went without one (#3128). Those rows are
// still reachable under Tasks, wearing a Plan badge.
type Facet = "" | "note" | "task" | "issue" | "spike" | "snippet" | "process";
// The facets that select TASKS. Kinds are subsets of `task`, so any of them
// means the duplicate report should be comparing tasks.
const TASK_FACETS = new Set<Facet>(["task", "issue", "spike"]);
const FACET_CHIPS: [Exclude<Facet, "">, string][] = [
["note", "Notes"],
["task", "Tasks"],
["issue", "Issues"],
["spike", "Spikes"],
["snippet", "Snippets"],
["process", "Processes"],
];
// ─── Filter state ───────────────────────────────────────────────────────────── // ─── Filter state ─────────────────────────────────────────────────────────────
const activeType = ref<"" | "note" | "task" | "plan" | "process">(""); const activeType = ref<Facet>("");
const activeTag = ref(""); const activeTag = ref("");
const sortMode = ref<"modified" | "created" | "alpha" | "type">("modified"); const sortMode = ref<"modified" | "created" | "alpha" | "type">("modified");
const searchQuery = ref(""); const searchQuery = ref("");
@@ -70,9 +95,10 @@ const dupGroups = ref<DupGroup[]>([]);
const dupSuggestion = ref(""); const dupSuggestion = ref("");
const dupLoading = ref(false); const dupLoading = ref(false);
const dupChecked = ref(false); const dupChecked = ref(false);
// The report follows the type filter: viewing tasks checks tasks. Anything // The report follows the type filter: viewing tasks — under ANY task facet,
// else (all / plan / process) checks notes — the kind with the most to find. // including a single kind — checks tasks. Everything else checks notes, the
const dupKind = computed(() => (activeType.value === "task" ? "task" : "note")); // kind with the most to find.
const dupKind = computed(() => (TASK_FACETS.has(activeType.value) ? "task" : "note"));
async function loadDuplicates() { async function loadDuplicates() {
dupLoading.value = true; dupLoading.value = true;
@@ -96,8 +122,11 @@ watch(dupKind, () => { dupChecked.value = false; dupGroups.value = []; });
// ─── Type counts ────────────────────────────────────────────────────────────── // ─── Type counts ──────────────────────────────────────────────────────────────
interface KnowledgeCounts { note: number; task: number; plan: number; process: number; total: number } // One number per facet, plus the grand total. Partial because the server sends
const typeCounts = ref<KnowledgeCounts>({ note: 0, task: 0, plan: 0, process: 0, total: 0 }); // a key only for a facet it has rows for. Kinds are subsets of `task` and are
// deliberately absent from `total` — including them would count an issue twice.
type KnowledgeCounts = Partial<Record<Exclude<Facet, "">, number>> & { total: number };
const typeCounts = ref<KnowledgeCounts>({ total: 0 });
async function fetchCounts() { async function fetchCounts() {
try { try {
@@ -274,9 +303,18 @@ function isOverdue(item: KnowledgeItem): boolean {
return new Date(item.due_date) < new Date(new Date().toDateString()); return new Date(item.due_date) < new Date(new Date().toDateString());
} }
// Each record kind opens in ITS OWN editor. A snippet used to fall through to
// /notes/:id, whose save is a plain PATCH of the body — which left the snippet's
// derived `data` mirror describing the previous version (#3128). The service now
// recomposes the mirror either way, so this is no longer the guard; it is simply
// that the note editor cannot edit a snippet's signature, language or locations,
// and offering it as the way in was always wrong. Processes stay here on
// purpose: they have no editor of their own and the note editor knows the type.
function openItem(item: KnowledgeItem) { function openItem(item: KnowledgeItem) {
if (item.note_type === 'task') { if (item.note_type === 'task') {
router.push(`/tasks/${item.id}`); router.push(`/tasks/${item.id}`);
} else if (item.note_type === 'snippet') {
router.push(`/snippets/${item.id}`);
} else { } else {
router.push(`/notes/${item.id}`); router.push(`/notes/${item.id}`);
} }
@@ -380,14 +418,14 @@ onUnmounted(() => {
<span v-if="typeCounts.total > 1" class="filter-count">{{ typeCounts.total }}</span> <span v-if="typeCounts.total > 1" class="filter-count">{{ typeCounts.total }}</span>
</button> </button>
<button <button
v-for="[val, label, key] in ([['note','Notes','note'],['task','Tasks','task'],['plan','Plans','plan'],['process','Processes','process']] as [string,string,string][])" v-for="[val, label] in FACET_CHIPS"
:key="val" :key="val"
class="filter-btn" class="filter-btn"
:class="{ active: activeType === val }" :class="{ active: activeType === val }"
@click="activeType = (val as '' | 'note' | 'task' | 'plan' | 'process')" @click="activeType = val"
> >
<span class="filter-btn-label">{{ label }}</span> <span class="filter-btn-label">{{ label }}</span>
<span v-if="typeCounts[key as keyof KnowledgeCounts] > 1" class="filter-count">{{ typeCounts[key as keyof KnowledgeCounts] }}</span> <span v-if="(typeCounts[val] ?? 0) > 1" class="filter-count">{{ typeCounts[val] }}</span>
</button> </button>
</div> </div>
@@ -501,6 +539,7 @@ onUnmounted(() => {
<span v-if="item.note_type === 'note'">Note</span> <span v-if="item.note_type === 'note'">Note</span>
<span v-else-if="item.note_type === 'task'">{{ item.task_kind === 'plan' ? 'Plan' : 'Task' }}</span> <span v-else-if="item.note_type === 'task'">{{ item.task_kind === 'plan' ? 'Plan' : 'Task' }}</span>
<span v-else-if="item.note_type === 'process'">Process</span> <span v-else-if="item.note_type === 'process'">Process</span>
<span v-else-if="item.note_type === 'snippet'">Snippet</span>
</span> </span>
<!-- Kind sits BESIDE the type badge, not inside it: the type badge <!-- Kind sits BESIDE the type badge, not inside it: the type badge
speaks the vocabulary of this view's type filter (note / task / speaks the vocabulary of this view's type filter (note / task /
@@ -881,6 +920,14 @@ onUnmounted(() => {
.badge--note { background: color-mix(in srgb, var(--fs-accent) 15%, transparent); color: #7A6DA8; } .badge--note { background: color-mix(in srgb, var(--fs-accent) 15%, transparent); color: #7A6DA8; }
.badge--task { background: rgba(212,160,23,0.15); color: #fbbf24; } .badge--task { background: rgba(212,160,23,0.15); color: #fbbf24; }
.badge--plan { background: rgba(99,102,241,0.18); color: #818cf8; } .badge--plan { background: rgba(99,102,241,0.18); color: #818cf8; }
/* Snippet and process are NEUTRAL on purpose. Both were unstyled — and the
snippet had no label either, so all 90 of them rendered an empty chip in
this feed (#3128). Giving them hues would put a third and fourth colour
beside KindBadge's warm/cool pair on the same card; a record type that
isn't an alarm reads better as plain. Standard body pair, so the contrast
is the one the palette already guarantees. */
.badge--snippet,
.badge--process { background: var(--fs-surface-raised); color: var(--fs-text-secondary); }
.k-card-body { flex: 1; padding-right: 40px; } .k-card-body { flex: 1; padding-right: 40px; }
.k-card-title { .k-card-title {
+5 -2
View File
@@ -94,8 +94,11 @@ class Note(Base, TimestampMixin, SoftDeleteMixin):
# name/language/signature/locations live here so they can be INDEXED. The # name/language/signature/locations live here so they can be INDEXED. The
# body keeps the same facts in readable markdown and remains what gets # body keeps the same facts in readable markdown and remains what gets
# embedded; this is a mirror for querying, not the source of truth for # embedded; this is a mirror for querying, not the source of truth for
# display. NULL on every row written before migration 0070, so readers fall # display — and it is DERIVED, so every path that writes a snippet's body
# back to parsing the body (see services/snippets.snippet_fields). # rewrites it too (services/snippets.recompose_data, called from
# notes.update_note). 0070 left it NULL on existing rows and
# snippets.backfill_snippet_data filled them at startup; readers still fall
# back to parsing the body when it is absent (snippet_fields).
data: Mapped[dict | None] = mapped_column(JSONB, nullable=True) data: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
__table_args__ = ( __table_args__ = (
+12 -4
View File
@@ -1,4 +1,4 @@
"""Unified Knowledge endpoint — notes, tasks, plans, and processes in one queryable feed.""" """Unified Knowledge endpoint — every record kind in one queryable feed."""
import logging import logging
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
@@ -6,12 +6,18 @@ from quart import Blueprint, jsonify, request
from scribe.auth import get_current_user_id, login_required from scribe.auth import get_current_user_id, login_required
from scribe.routes.utils import parse_pagination from scribe.routes.utils import parse_pagination
from scribe.services.access import label_shared_items from scribe.services.access import label_shared_items
from scribe.services.knowledge import FACET_TYPES
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
knowledge_bp = Blueprint("knowledge", __name__, url_prefix="/api/knowledge") knowledge_bp = Blueprint("knowledge", __name__, url_prefix="/api/knowledge")
_VALID_TYPES = {"note", "task", "plan", "process"} # Derived from the service's facet table, never re-listed here. This set was a
# hand-kept copy and had drifted three kinds behind it: it admitted `plan`
# (retired in 0066) and rejected `issue` (shipped in 0065, 435 rows) and
# `snippet` — so the browse surface could not filter to the kinds it was
# already rendering badges for (#3128).
_VALID_TYPES = FACET_TYPES
_VALID_SORTS = {"modified", "created", "alpha", "type"} _VALID_SORTS = {"modified", "created", "alpha", "type"}
@@ -21,7 +27,9 @@ async def list_knowledge():
"""Return paginated knowledge objects with optional filtering. """Return paginated knowledge objects with optional filtering.
Query params: Query params:
type — one of note|task|plan|process (omit for all) type — a facet from services.knowledge._FACETS: a record type
(note|process|snippet) or a task kind (task for any,
else work|issue|spike|plan). Omit for all.
tags — comma-separated tag filter (AND logic) tags — comma-separated tag filter (AND logic)
sort — modified|created|alpha|type (default: modified) sort — modified|created|alpha|type (default: modified)
q — search query (semantic when provided, keyword fallback) q — search query (semantic when provided, keyword fallback)
@@ -127,7 +135,7 @@ async def get_knowledge_batch():
@knowledge_bp.route("/tags", methods=["GET"]) @knowledge_bp.route("/tags", methods=["GET"])
@login_required @login_required
async def list_knowledge_tags(): async def list_knowledge_tags():
"""Return all tags used across knowledge objects (excludes tasks).""" """Return all tags used across knowledge objects, narrowed to one facet."""
uid = get_current_user_id() uid = get_current_user_id()
note_type = request.args.get("type", "").strip().lower() or None note_type = request.args.get("type", "").strip().lower() or None
+116 -59
View File
@@ -1,4 +1,4 @@
"""Knowledge service — unified query across notes, tasks, plans, and processes. """Knowledge service — one query across every record kind Scribe holds.
ACL (rules #47/#78, decision note 2094): these queries were owner-only until ACL (rules #47/#78, decision note 2094): these queries were owner-only until
2026-07-25, which meant a record shared with you could be opened by id but never 2026-07-25, which meant a record shared with you could be opened by id but never
@@ -265,22 +265,94 @@ def _note_to_item(note: Note) -> dict:
return item return item
def _apply_type_filter(stmt, note_type: str | None): # What each type facet MEANS, once, for every arm that has to know.
"""Apply the type facet to a Note select. #
# The vocabulary spans BOTH typing axes — `note_type` for non-task records and
# `task_kind` for tasks — so a facet cannot be a filter on one column, which is
# why this is a table rather than a chain of ifs. Each entry is
# (is_task, the value pinned on that axis); None pins nothing, i.e. every task.
#
# It is a table because the alternative had already gone wrong. The predicate
# was written three times — a SQL if-chain, a Python if-chain over semantic
# candidates, and a ternary computing the `is_task` pre-filter — and the three
# only agreed by luck. Adding `issue` to the SQL arm alone (the obvious edit,
# and the one #3128 was about to make) would have set the pre-filter to
# is_task=False, handed the Python arm a candidate set containing no tasks at
# all, and returned an empty semantic half for the Issues facet forever, with
# nothing red anywhere. A new facet is now one row here.
#
# `plan` is retired (0066) but kept: 90 legacy plan-tasks exist and a facet
# they answer to costs one line. It simply has no chip in the UI any more.
_FACETS: dict[str, tuple[bool, str | None]] = {
"task": (True, None),
"work": (True, "work"),
"issue": (True, "issue"),
"spike": (True, "spike"),
"plan": (True, "plan"),
"note": (False, "note"),
"process": (False, "process"),
"snippet": (False, "snippet"),
}
'task' = any task (status not null); 'plan' = a task with task_kind='plan'; # The non-task record types, for the counts query. Derived so it cannot drift
any other non-empty type = a non-task note of that note_type; None = all. # from the table above.
NON_TASK_FACETS = tuple(
value for _is_task, value in _FACETS.values() if not _is_task and value
)
Trashed rows (deleted_at set) are always excluded. # The whole vocabulary, for the door's request validation — public so the route
# validates against the same table the query reads instead of a hand-kept copy.
FACET_TYPES = frozenset(_FACETS)
# An unrecognised facet resolves to "a non-task note whose note_type is that
# string" — which matches nothing, since no row stores an unknown type. That is
# the behaviour the old if-chain had by falling through, and it is the right
# one: a typo should return an empty list, never the whole corpus.
def _facet(note_type: str) -> tuple[bool, str | None]:
return _FACETS.get(note_type, (False, note_type))
def facet_is_task(note_type: str | None) -> bool | None:
"""The `is_task` pre-filter a facet implies — None when it spans both.
Used to narrow the semantic candidate set before it is fetched. Reads the
same table `_apply_type_filter` and `matches_facet` read, so the pre-filter
can no longer disagree with the predicate it is meant to anticipate.
""" """
if not note_type:
return None
return _facet(note_type)[0]
def matches_facet(note, note_type: str | None) -> bool:
"""The Python dialect of `_apply_type_filter`, for candidates the vector
search has already fetched — there is no query left to narrow.
Generated from the same table, so this is a translation rather than a
second implementation. Note the `not note.is_task` arm: the hand-written
version omitted it and was saved only by the upstream pre-filter.
"""
if not note_type:
return True
is_task, value = _facet(note_type)
if is_task:
return note.is_task and (value is None or note.task_kind == value)
return not note.is_task and note.note_type == value
def _apply_type_filter(stmt, note_type: str | None):
"""Apply the type facet to a Note select. Trashed rows are always excluded."""
stmt = stmt.where(Note.deleted_at.is_(None)) stmt = stmt.where(Note.deleted_at.is_(None))
if note_type == "task": if not note_type:
return stmt.where(Note.status.isnot(None)) return stmt
if note_type == "plan": is_task, value = _facet(note_type)
return stmt.where(Note.status.isnot(None)).where(Note.task_kind == "plan") if is_task:
if note_type: stmt = stmt.where(Note.status.isnot(None))
return stmt.where(Note.note_type == note_type).where(Note.status.is_(None)) if value is not None:
return stmt stmt = stmt.where(Note.task_kind == value)
return stmt
return stmt.where(Note.status.is_(None)).where(Note.note_type == value)
async def query_knowledge( async def query_knowledge(
@@ -295,7 +367,7 @@ async def query_knowledge(
locations: dict[str, str] | None = None, locations: dict[str, str] | None = None,
verification: str = "", verification: str = "",
) -> tuple[list[dict], int]: ) -> tuple[list[dict], int]:
"""Query knowledge objects (non-task notes) with filters. """Query knowledge objects with filters.
`project_id` narrows to one project (None = every project). `project_id` narrows to one project (None = every project).
@@ -424,7 +496,7 @@ async def _semantic_knowledge_search(
INTERACTIVE_SEARCH_THRESHOLD, INTERACTIVE_SEARCH_THRESHOLD,
semantic_search_notes, semantic_search_notes,
) )
is_task_filter = True if note_type in ("task", "plan") else (False if note_type else None) is_task_filter = facet_is_task(note_type)
import time as _time import time as _time
_t0 = _time.perf_counter() _t0 = _time.perf_counter()
candidates = await semantic_search_notes( candidates = await semantic_search_notes(
@@ -454,11 +526,7 @@ async def _semantic_knowledge_search(
for _score, note in candidates: for _score, note in candidates:
if note.deleted_at is not None: if note.deleted_at is not None:
continue continue
if note_type == "task" and not note.is_task: if not matches_facet(note, note_type):
continue
elif note_type == "plan" and (not note.is_task or note.task_kind != "plan"):
continue
elif note_type and note_type not in ("task", "plan") and note.note_type != note_type:
continue continue
if tags and not all(t in (note.tags or []) for t in tags): if tags and not all(t in (note.tags or []) for t in tags):
continue continue
@@ -514,51 +582,40 @@ async def get_knowledge_counts(user_id: int, tags: list[str] | None = None) -> d
search would surface.""" search would surface."""
visible = browsable_notes_clause(user_id) visible = browsable_notes_clause(user_id)
async with async_session() as session: async with async_session() as session:
# Count non-task types def _scoped(stmt):
stmt = ( stmt = stmt.where(visible).where(Note.deleted_at.is_(None))
select(Note.note_type, func.count(Note.id)) for tag in tags or []:
.where(visible)
.where(Note.status.is_(None))
.where(Note.deleted_at.is_(None))
.where(Note.note_type.in_(["note", "process"]))
.group_by(Note.note_type)
)
if tags:
for tag in tags:
stmt = stmt.where(Note.tags.contains([tag])) stmt = stmt.where(Note.tags.contains([tag]))
rows = list((await session.execute(stmt)).all()) return stmt
counts = {row[0]: row[1] for row in rows}
# Count tasks separately (is_task = status IS NOT NULL) # One grouped query per typing axis. The task half used to be a count
task_stmt = ( # for 'task' plus a second count for 'plan', which is why 'issue' —
select(func.count(Note.id)) # 17% of every task here — had no number to show: each kind needed its
.where(visible) # own query and nobody added one. Grouping by task_kind counts every
# kind, including ones added later, for the same two round-trips.
non_task = _scoped(
select(Note.note_type, func.count(Note.id))
.where(Note.status.is_(None))
.where(Note.note_type.in_(NON_TASK_FACETS))
).group_by(Note.note_type)
counts = {t: n for t, n in (await session.execute(non_task)).all()}
by_kind = _scoped(
select(Note.task_kind, func.count(Note.id))
.where(Note.status.isnot(None)) .where(Note.status.isnot(None))
.where(Note.deleted_at.is_(None)) ).group_by(Note.task_kind)
) kind_counts = {k: n for k, n in (await session.execute(by_kind)).all()}
if tags:
for tag in tags:
task_stmt = task_stmt.where(Note.tags.contains([tag]))
task_count: int = (await session.execute(task_stmt)).scalar_one()
counts["task"] = task_count
# Plans are a subset of tasks (task_kind='plan'); counted for the facet # Kinds are SUBSETS of 'task' and are deliberately left out of the total —
# but NOT added to total to avoid double-counting against "task". # adding them would count every task twice.
plan_stmt = ( counts["task"] = sum(kind_counts.values())
select(func.count(Note.id)) for kind, value in _FACETS.items():
.where(visible) if value[0] and value[1] is not None:
.where(Note.status.isnot(None)) counts[kind] = kind_counts.get(kind, 0)
.where(Note.task_kind == "plan")
.where(Note.deleted_at.is_(None))
)
if tags:
for tag in tags:
plan_stmt = plan_stmt.where(Note.tags.contains([tag]))
counts["plan"] = (await session.execute(plan_stmt)).scalar_one()
for t in ("note", "task", "plan", "process"): for t in NON_TASK_FACETS:
counts.setdefault(t, 0) counts.setdefault(t, 0)
counts["total"] = sum(counts[t] for t in ("note", "task", "process")) counts["total"] = counts["task"] + sum(counts[t] for t in NON_TASK_FACETS)
return counts return counts
+24
View File
@@ -9,6 +9,13 @@ from scribe.models.note import Note, TaskKind, TaskPriority, TaskStatus
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# The fields `snippets.parse_snippet_fields` reads. Writing any of them can
# change what a snippet's derived `data` mirror should say, so update_note
# recomposes the mirror when one moves. Kept here as a set of NAMES rather
# than imported, because it describes update_note's own `fields` dict, not the
# parser's signature.
_PARSED_FROM_BODY = frozenset({"title", "body", "tags"})
def embed_note(note) -> None: def embed_note(note) -> None:
"""Refresh a note's embedding, fire-and-forget. """Refresh a note's embedding, fire-and-forget.
@@ -439,6 +446,23 @@ async def update_note(user_id: int, note_id: int, **fields: object) -> Note | No
elif key == "tags" and isinstance(value, list): elif key == "tags" and isinstance(value, list):
value = _normalize_tags(value) value = _normalize_tags(value)
setattr(note, key, value) setattr(note, key, value)
# A snippet's `data` is DERIVED from its body — so a write that moves
# the body through this generic door must move the mirror with it
# (#3128). Without this, PATCH /api/notes/<snippet_id> {body} left the
# mirror behind, and snippet_fields PREFERS the mirror: the row went on
# reporting its old repo/path/symbol to prior-art recall while showing
# its new body. `update_snippet` composes the mirror itself and passes
# it explicitly, so an explicit `data` always wins — the caller that
# knows the field set beats the one that can only re-read the body.
if "data" not in fields and not _PARSED_FROM_BODY.isdisjoint(fields):
# Imported here, not at module scope: services/snippets.py calls
# back into this module (update_snippet -> update_note), so a
# top-level import is a cycle.
from scribe.services.snippets import (
SNIPPET_NOTE_TYPE, recompose_data,
)
if note.note_type == SNIPPET_NOTE_TYPE:
note.data = recompose_data(note)
# Auto-set lifecycle timestamps on status transitions # Auto-set lifecycle timestamps on status transitions
if "status" in fields: if "status" in fields:
_now = datetime.now(timezone.utc) _now = datetime.now(timezone.utc)
+50 -4
View File
@@ -540,14 +540,60 @@ def compose_data(
return out return out
def recompose_data(note) -> dict:
"""Rebuild a snippet's `data` mirror from its own body, title and tags.
For the GENERIC note door. `update_snippet` composes the mirror itself from
the field set it just merged and never needs this; a plain
`update_note(body=...)` — which the Knowledge feed's editor issues, because
a snippet card there routes to /notes/:id — has no idea the mirror exists,
and left it stale. `snippet_fields` then PREFERS the stale mirror, so the
row reported its old repo/path/symbol to prior-art recall while displaying
its new body: confidently wrong, which is worse than no record (#3128).
The body is the authority; the mirror is derived. That is already the rule
this file states — it just had no enforcement on the path that bypasses
`update_snippet`.
`verification` and `provenance` are CARRIED, not recomposed, because
neither is in the body to parse — the same carry `compose_data` does for
the snippet service's own writes. Note that a verdict does not need
invalidating here: `code_sha` is recomputed from the new code, so a stale
verdict expires itself on read exactly as it does after any other edit.
"""
parsed = parse_snippet_fields(note.title, note.body, note.tags)
prior = note.data or {}
return compose_data(
name=parsed["name"],
when_to_use=parsed["when_to_use"],
signature=parsed["signature"],
language=parsed["language"],
code=parsed["code"],
locations=parsed["locations"],
merged_from=parsed["merged_from"],
verification=prior.get("verification"),
provenance=prior.get("provenance"),
)
def snippet_fields(note) -> dict: def snippet_fields(note) -> dict:
"""Structured fields for a snippet, preferring the indexed `data` column and """Structured fields for a snippet, preferring the indexed `data` column and
falling back to parsing the body. falling back to parsing the body.
Both paths must agree, because rows written before migration 0070 have no THE BODY IS THE AUTHORITY; `data` is a mirror derived from it. Every writer
`data` and are never backfilled — a hand-edited body is the authority for keeps them in step — the snippet service composes the mirror from the field
those, and there is no deadline by which they must be converted. `code` only set it just merged, `update_note` recomposes it when a body reaches the
ever comes from the body, since `data` doesn't carry it. generic door (#3128), and `backfill_snippet_data` filled the pre-0070 rows
at startup. The fallback below is therefore a belt to that braces, not a
second source of truth: it is what a row looks like before the backfill has
run, and it must keep agreeing with the mirror.
(This docstring used to say those rows were "never backfilled". That was
true when 0070 landed and stopped being true when the backfill shipped; it
is corrected here because the sentence read as licence for a stale mirror,
which is exactly the bug #3128 found.)
`code` only ever comes from the body, since `data` doesn't carry it.
""" """
parsed = parse_snippet_fields(note.title, note.body, note.tags) parsed = parse_snippet_fields(note.title, note.body, note.tags)
stored = getattr(note, "data", None) stored = getattr(note, "data", None)
+10
View File
@@ -12,6 +12,16 @@ from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock from unittest.mock import AsyncMock, MagicMock
def compiled_sql(element) -> str:
"""A SQLAlchemy clause or statement rendered as literal SQL text.
For asserting on the shape of a predicate without a database — which is how
the visibility clauses and the knowledge facets are both tested. Was a
private copy in each of those modules before #3128 needed a third.
"""
return str(element.compile(compile_kwargs={"literal_binds": True}))
def make_mock_session() -> AsyncMock: def make_mock_session() -> AsyncMock:
"""A stand-in for ``async_session()`` — usable as ``async with``, with the """A stand-in for ``async_session()`` — usable as ``async with``, with the
commit/refresh/add surface a service touches. commit/refresh/add surface a service touches.
+120
View File
@@ -0,0 +1,120 @@
"""The type facet, in both of its dialects.
The facet predicate is written twice by necessity — as SQL for rows the
database hands back, and as Python for candidates the vector search has
already fetched — plus a third time as the `is_task` pre-filter that narrows
the candidate set before it exists. Those three used to be hand-written and
agreed only by luck: adding `issue` to the SQL arm alone would have set the
pre-filter to is_task=False, handed the Python arm a candidate set with no
tasks in it, and returned an empty semantic half for the Issues facet forever
with nothing red anywhere (#3128).
They are now generated from one table. These tests pin the property that made
the trap possible, so it cannot come back by a different route.
"""
import pytest
from scribe.services.knowledge import (
_FACETS,
FACET_TYPES,
NON_TASK_FACETS,
_apply_type_filter,
facet_is_task,
matches_facet,
)
from tests.helpers import compiled_sql, fake_note, fake_snippet, fake_task
def _sql(note_type):
from sqlalchemy import select
from scribe.models.note import Note
return compiled_sql(_apply_type_filter(select(Note.id), note_type))
# One representative row per shape the corpus actually holds.
ROWS = {
"plain note": fake_note(note_type="note"),
"process": fake_note(note_type="process"),
"snippet": fake_snippet(),
"work task": fake_task(task_kind="work", note_type="note"),
"issue": fake_task(task_kind="issue", note_type="note"),
"spike": fake_task(task_kind="spike", note_type="note"),
"legacy plan": fake_task(task_kind="plan", note_type="note"),
}
@pytest.mark.parametrize("facet", sorted(FACET_TYPES))
def test_pre_filter_never_excludes_a_row_the_facet_wants(facet):
"""THE regression. `facet_is_task` narrows the semantic candidate set before
`matches_facet` ever sees it, so a pre-filter that disagrees with the
predicate doesn't return wrong rows — it returns NO rows, silently, on one
half of a hybrid search."""
want = facet_is_task(facet)
for label, row in ROWS.items():
if matches_facet(row, facet):
assert want is None or want == row.is_task, (
f"facet {facet!r} accepts the {label} row, but its pre-filter "
f"asks for is_task={want} while the row has is_task={row.is_task} "
f"— the semantic arm would never be handed this row"
)
@pytest.mark.parametrize(
"facet,expected",
[
("task", {"work task", "issue", "spike", "legacy plan"}),
("issue", {"issue"}),
("spike", {"spike"}),
("work", {"work task"}),
("plan", {"legacy plan"}),
("note", {"plain note"}),
("process", {"process"}),
("snippet", {"snippet"}),
("", set(ROWS)),
],
)
def test_each_facet_selects_exactly_its_own_rows(facet, expected):
assert {k for k, row in ROWS.items() if matches_facet(row, facet)} == expected
def test_a_plain_note_is_not_selected_by_its_own_type_when_it_is_a_task():
"""A task's `note_type` is 'note' — that column says nothing about task-ness.
The hand-written Python arm omitted the `status IS NULL` half its SQL twin
carried, so it only avoided returning every task under the Notes facet
because the pre-filter had already dropped them."""
assert matches_facet(fake_task(note_type="note"), "note") is False
def test_an_unknown_facet_matches_nothing_rather_than_everything():
"""A typo must return an empty list, never the whole corpus."""
assert all(not matches_facet(row, "wrok") for row in ROWS.values())
assert "notes.status IS NULL" in _sql("wrok")
def test_the_live_task_kinds_are_all_facets():
"""`issue` shipped in 0065 and `spike` in 0091; the browse vocabulary went
three kinds without noticing either."""
for kind in ("work", "issue", "spike"):
assert kind in FACET_TYPES and _FACETS[kind][0] is True
def test_non_task_facets_are_the_note_types_and_only_those():
assert set(NON_TASK_FACETS) == {"note", "process", "snippet"}
@pytest.mark.parametrize("facet", sorted(FACET_TYPES))
def test_sql_arm_constrains_the_axis_the_facet_lives_on(facet):
"""The SQL dialect of the same table. A task facet must pin `status` (and,
for a single kind, `task_kind`); a record-type facet must pin `note_type`
AND exclude tasks."""
sql = _sql(facet)
is_task, value = _FACETS[facet]
assert "notes.deleted_at IS NULL" in sql
if is_task:
assert "notes.status IS NOT NULL" in sql
assert (f"notes.task_kind = '{value}'" in sql) is (value is not None)
else:
assert "notes.status IS NULL" in sql
assert f"notes.note_type = '{value}'" in sql
+2 -3
View File
@@ -20,10 +20,9 @@ from scribe.services.access import (
notes_visibility_clause, notes_visibility_clause,
readable_notes_clause, readable_notes_clause,
) )
from tests.helpers import compiled_sql
_sql = compiled_sql
def _sql(clause) -> str:
return str(clause.compile(compile_kwargs={"literal_binds": True}))
def _read(user_id: int = 7) -> str: def _read(user_id: int = 7) -> str:
+69 -22
View File
@@ -1,4 +1,12 @@
"""get_knowledge_counts includes the 'process' type and counts it in total.""" """get_knowledge_counts — one number per facet, and an honest total.
Two grouped queries, one per typing axis. It used to be three: a grouped count
over note_type restricted to ("note", "process"), a scalar count of tasks, and
a second scalar just for plans. That shape is why `issue` had no number —
every kind needed a query of its own and nobody added one — and why the "All"
chip sat ~90 below the list it labelled, since snippets were in the feed but
in no count (#3128).
"""
from unittest.mock import AsyncMock, MagicMock, patch from unittest.mock import AsyncMock, MagicMock, patch
import pytest import pytest
@@ -11,29 +19,68 @@ def _grouped(rows):
return r return r
def _scalar(n): async def _counts(non_task_rows, kind_rows, **kwargs):
r = MagicMock()
r.scalar_one.return_value = n
return r
@pytest.mark.asyncio
async def test_counts_include_process_in_facet_and_total():
session = make_mock_session() session = make_mock_session()
# 1) grouped non-task counts, 2) task count, 3) plan count # 1) non-task rows grouped by note_type, 2) task rows grouped by task_kind
session.execute = AsyncMock(side_effect=[ session.execute = AsyncMock(
_grouped([("note", 3), ("process", 2)]), side_effect=[_grouped(non_task_rows), _grouped(kind_rows)]
_scalar(1), # tasks )
_scalar(0), # plans
])
with patch("scribe.services.knowledge.async_session") as cls: with patch("scribe.services.knowledge.async_session") as cls:
cls.return_value = session cls.return_value = session
from scribe.services.knowledge import get_knowledge_counts from scribe.services.knowledge import get_knowledge_counts
counts = await get_knowledge_counts(user_id=1) return await get_knowledge_counts(user_id=1, **kwargs), session
assert counts["process"] == 2
# facet keys all present (setdefault) @pytest.mark.asyncio
for key in ("note", "task", "plan", "process"): async def test_every_facet_gets_a_number_including_the_kinds():
assert key in counts counts, _ = await _counts(
# total = note(3) + task(1) + process(2) [("note", 395), ("process", 3), ("snippet", 90)],
assert counts["total"] == 6 [("work", 2104), ("issue", 435), ("spike", 1), ("plan", 90)],
)
assert counts["note"] == 395
assert counts["process"] == 3
assert counts["snippet"] == 90
assert counts["issue"] == 435
assert counts["spike"] == 1
assert counts["plan"] == 90
assert counts["work"] == 2104
@pytest.mark.asyncio
async def test_task_is_the_sum_of_its_kinds():
"""`task` is not counted separately any more — it is what the kinds add up
to, so the two can't disagree."""
counts, _ = await _counts([], [("work", 2104), ("issue", 435), ("spike", 1)])
assert counts["task"] == 2540
@pytest.mark.asyncio
async def test_total_counts_snippets_and_counts_no_task_twice():
"""The All chip labels a feed that contains every kind, so it has to count
every kind — and exactly once. Kinds are subsets of `task`; adding them
would count each issue a second time."""
counts, _ = await _counts(
[("note", 10), ("process", 2), ("snippet", 5)],
[("work", 20), ("issue", 4)],
)
assert counts["task"] == 24
assert counts["total"] == 10 + 2 + 5 + 24
@pytest.mark.asyncio
async def test_absent_facets_report_zero_rather_than_missing():
counts, _ = await _counts([("note", 1)], [])
assert counts["note"] == 1
for key in ("process", "snippet", "task", "work", "issue", "spike", "plan"):
assert counts[key] == 0, key
assert counts["total"] == 1
@pytest.mark.asyncio
async def test_a_tag_filter_narrows_both_axes():
"""A tag has to reach both queries, or the chips would disagree with each
other under a filter — tasks narrowed, notes not."""
_, session = await _counts([], [], tags=["python"])
assert session.execute.await_count == 2
for call in session.execute.await_args_list:
assert "notes.tags" in str(call.args[0])
+111
View File
@@ -0,0 +1,111 @@
"""A snippet's `data` mirror survives the GENERIC note door.
`notes.data` is derived from the body. The snippet service always composed it
from the field set it had just merged, so `update_snippet` was never the
problem — the problem was every other way a snippet's body could be written.
`update_note` is a `hasattr` loop with no snippet awareness, and both doors
reach it: PATCH /api/notes/<id> and the MCP update_note tool. The Knowledge
feed handed you that path, because a snippet card there routed to /notes/:id.
The failure was silent and the wrong way round: `snippet_fields` PREFERS the
mirror, so the row went on reporting its old repo/path/symbol to the location
reverse lookup and to prior-art recall while displaying its new body — a record
surfaced with full authority and wrong, which the drift-check docstring calls
worse than having no record at all (#3128).
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from tests.helpers import fake_note, fake_snippet, make_mock_session
OLD_MIRROR = {
"name": "debounce",
"language": "javascript",
"locations": [{"repo": "Scribe", "path": "old/place.js", "symbol": "debounce"}],
"verification": {"status": "ok", "code_sha": "abc", "checked_at": "2026-01-01"},
"provenance": {"commit_sha": "deadbeef"},
}
MOVED_BODY = (
"**Locations:**\n"
"- `Scribe` · `new/place.ts` · `debounce`\n\n"
"```typescript\nexport const debounce = 1;\n```\n"
)
async def _update(note, **fields):
session = make_mock_session()
result = MagicMock()
result.scalars.return_value.first.return_value = note
session.execute = AsyncMock(return_value=result)
with patch("scribe.services.notes.async_session") as cls, \
patch("scribe.services.notes.embed_note", MagicMock()), \
patch("scribe.services.notes._maybe_reactivate_project", AsyncMock()), \
patch("scribe.services.note_versions.create_version", AsyncMock()):
cls.return_value = session
from scribe.services.notes import update_note
await update_note(user_id=7, note_id=note.id, **fields)
return note
@pytest.mark.asyncio
async def test_a_body_write_moves_the_mirror_with_it():
note = fake_snippet(data=dict(OLD_MIRROR), project_id=None)
await _update(note, body=MOVED_BODY)
assert note.data["locations"] == [
{"repo": "Scribe", "path": "new/place.ts", "symbol": "debounce"}
], "the mirror still describes where the snippet used to live"
assert note.data["language"] == "typescript"
@pytest.mark.asyncio
async def test_the_verdict_and_provenance_are_carried_not_dropped():
"""Neither is in the body to parse, so recomposing must carry them. An
ordinary edit must not erase the last drift check — and it needs no
invalidation branch either: `code_sha` is recomputed from the new code, so
a verdict stamped against the old code expires itself on read."""
note = fake_snippet(data=dict(OLD_MIRROR), project_id=None)
await _update(note, body=MOVED_BODY)
assert note.data["verification"] == OLD_MIRROR["verification"]
assert note.data["provenance"] == OLD_MIRROR["provenance"]
assert note.data["code_sha"] != OLD_MIRROR["verification"]["code_sha"]
@pytest.mark.asyncio
async def test_an_explicit_data_wins_over_recomposition():
"""`update_snippet` composes the mirror from the merged field set it holds
and passes it here. That caller knows things the body cannot be re-read for
— which locations were replaced, whether provenance survives the edit — so
an explicit mirror must not be recomputed out from under it."""
note = fake_snippet(data=dict(OLD_MIRROR), project_id=None)
authoritative = {"name": "from the service", "locations": []}
await _update(note, body=MOVED_BODY, data=authoritative)
assert note.data == authoritative
@pytest.mark.asyncio
async def test_a_plain_note_is_left_alone():
"""Only snippets carry a mirror; a note's `data` must not be invented."""
note = fake_note(note_type="note", data=None, project_id=None)
await _update(note, body="just some prose")
assert note.data is None
@pytest.mark.asyncio
async def test_a_write_that_cannot_change_the_parse_does_not_touch_the_mirror():
"""Status, priority, project — none of them is an input to the body parser,
so recomposing on them would be work for nothing and would rebuild a mirror
from a body nobody claimed to have changed."""
note = fake_snippet(data=dict(OLD_MIRROR), project_id=None)
await _update(note, project_id=4)
assert note.data == OLD_MIRROR
@pytest.mark.asyncio
async def test_a_title_change_reaches_the_mirror_too():
"""A snippet's NAME lives in its title, not its body — `parse_snippet_fields`
reads both, so both are triggers."""
note = fake_snippet(data=dict(OLD_MIRROR), project_id=None)
await _update(note, title="throttle — cap a callback's rate")
assert note.data["name"] == "throttle"
assert note.data["when_to_use"] == "cap a callback's rate"