feat(snippets): near-duplicate finder — surface the sets worth merging
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 14s
CI & Build / integration (push) Successful in 36s
CI & Build / Python tests (push) Successful in 55s
CI & Build / Build & push image (push) Successful in 44s

#231's premise was unifying reusable things already scattered as one-offs.
The create gate PREVENTS a new duplicate and merge_snippets CURES one you
point it at, but nothing FOUND the duplicates already in the record —
someone had to notice them by hand, which is the exact failure the Drafter
exists to remove.

One indexed self-join over note_embeddings, not an N² Python scan:
pgvector's cosine distance is the same operator semantic search uses, so a
similarity floor is a distance ceiling and the work stays in Postgres.
`left.note_id < right.note_id` yields each unordered pair once and drops
the self-pair that would otherwise dominate the ranking.

Pairs are collapsed into merge SETS by connected components. Transitive on
purpose: A~B plus B~C puts all three together even when A and C don't
directly clear the bar, which is what merge actually does (it folds every
source into one survivor). The cost is that a chain of mild resemblances
can rope in a member that isn't really alike — so the UI presents a set as
a proposal, shows the members, and never merges without a confirm.

Two scope decisions worth naming:

- OWN snippets only. merge_snippets requires one owner across the set, so
  surfacing someone else's would propose a merge that cannot be performed.
  The report is bounded by what the operator can act on, not what they can
  see.

- Threshold defaults to 0.82, LOOSER than the write gate's 0.90, and is a
  setting rather than a constant (rule #25). The gate blocks a create and
  has to be unforgiving of noise; this only suggests a merge under review,
  so it must reach further or it would never surface the pairs the gate
  already let through — which are precisely the ones that accumulated.

Fixes a real bug in the merge flow while wiring the UI: selectedList
filtered the selection against the CURRENT PAGE, and doMerge derives its
source ids from that list. A corpus-wide suggested group with off-page
members would have rendered incomplete and silently merged only the
visible subset. A group under review is now the authority for that list.

Refs #2088

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-28 18:32:40 -04:00
co-authored by Claude Opus 5
parent 84c5c0dc81
commit 6db791965f
10 changed files with 547 additions and 8 deletions
+19
View File
@@ -154,6 +154,25 @@ export async function deleteSnippet(id: number): Promise<void> {
return apiDelete(`/api/snippets/${id}`);
}
/** A set of snippets that resemble each other closely enough to be worth
* merging. Grouping is transitive, so a set can hold members that don't
* directly resemble each other — read it as a proposal, not a verdict. */
export interface DuplicateGroup {
note_ids: number[];
snippets: { id: number; title: string }[];
/** The strongest resemblance within the set — how confident the suggestion is. */
top_score: number;
}
/** Near-duplicates already in the record. The create gate prevents new ones and
* merge cures the ones you point it at; this is what finds them. */
export async function findDuplicateSnippets(
threshold?: number,
): Promise<{ groups: DuplicateGroup[]; threshold: number }> {
const qs = threshold ? `?threshold=${threshold}` : "";
return apiGet(`/api/snippets/duplicates${qs}`);
}
/** Record a drift-check verdict. The check itself runs where the code is — an
* agent with the working tree — since Scribe has no checkout. This stores what
* was found, and is how the UI clears a stale marker after a manual fix. */