feat(snippets): near-duplicate finder — surface the sets worth merging
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 14s
CI & Build / integration (push) Successful in 36s
CI & Build / Python tests (push) Successful in 55s
CI & Build / Build & push image (push) Successful in 44s

#231's premise was unifying reusable things already scattered as one-offs.
The create gate PREVENTS a new duplicate and merge_snippets CURES one you
point it at, but nothing FOUND the duplicates already in the record —
someone had to notice them by hand, which is the exact failure the Drafter
exists to remove.

One indexed self-join over note_embeddings, not an N² Python scan:
pgvector's cosine distance is the same operator semantic search uses, so a
similarity floor is a distance ceiling and the work stays in Postgres.
`left.note_id < right.note_id` yields each unordered pair once and drops
the self-pair that would otherwise dominate the ranking.

Pairs are collapsed into merge SETS by connected components. Transitive on
purpose: A~B plus B~C puts all three together even when A and C don't
directly clear the bar, which is what merge actually does (it folds every
source into one survivor). The cost is that a chain of mild resemblances
can rope in a member that isn't really alike — so the UI presents a set as
a proposal, shows the members, and never merges without a confirm.

Two scope decisions worth naming:

- OWN snippets only. merge_snippets requires one owner across the set, so
  surfacing someone else's would propose a merge that cannot be performed.
  The report is bounded by what the operator can act on, not what they can
  see.

- Threshold defaults to 0.82, LOOSER than the write gate's 0.90, and is a
  setting rather than a constant (rule #25). The gate blocks a create and
  has to be unforgiving of noise; this only suggests a merge under review,
  so it must reach further or it would never surface the pairs the gate
  already let through — which are precisely the ones that accumulated.

Fixes a real bug in the merge flow while wiring the UI: selectedList
filtered the selection against the CURRENT PAGE, and doMerge derives its
source ids from that list. A corpus-wide suggested group with off-page
members would have rendered incomplete and silently merged only the
visible subset. A group under review is now the authority for that list.

Refs #2088

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-28 18:32:40 -04:00
parent 84c5c0dc81
commit 6db791965f
10 changed files with 547 additions and 8 deletions
+156 -6
View File
@@ -1,7 +1,13 @@
<script setup lang="ts">
import { ref, computed, onMounted } from "vue";
import { useRouter } from "vue-router";
import { listSnippets, mergeSnippets, type SnippetListItem } from "@/api/snippets";
import {
findDuplicateSnippets,
listSnippets,
mergeSnippets,
type DuplicateGroup,
type SnippetListItem,
} from "@/api/snippets";
import { useToastStore } from "@/stores/toast";
const router = useRouter();
@@ -50,13 +56,54 @@ const showMergeModal = ref(false);
const canonicalId = ref<number | null>(null);
const merging = ref(false);
const selectedList = computed(() =>
snippets.value.filter((s) => selectedIds.value.has(s.id)),
);
// Near-duplicate report (#2088). Loaded on demand, not with the list: it's a
// pairwise scan and most visits to this page aren't a tidy-up.
const duplicateGroups = ref<DuplicateGroup[]>([]);
const dupLoading = ref(false);
const dupChecked = ref(false);
// Set while merging a SUGGESTED group. The report reaches the whole corpus, so
// its members need not all be on the current page — see selectedList.
const reviewingGroup = ref<DuplicateGroup | null>(null);
/** The records the merge modal acts on.
*
* Normally that's the selection filtered against what's on screen. But a
* suggested group is corpus-wide: filtering it by the current page would render
* an incomplete set AND silently narrow what doMerge folds in, since it derives
* its source ids from this list. When a group is under review it is the
* authority. */
const selectedList = computed<{ id: number; title: string }[]>(() => {
if (reviewingGroup.value) return reviewingGroup.value.snippets;
return snippets.value.filter((s) => selectedIds.value.has(s.id));
});
async function loadDuplicates() {
dupLoading.value = true;
try {
const data = await findDuplicateSnippets();
duplicateGroups.value = data.groups;
dupChecked.value = true;
} catch {
toast.show("Couldn't check for duplicates", "error");
} finally {
dupLoading.value = false;
}
}
/** Hand a suggested group to the existing merge flow, pre-selected. The operator
* still picks which record survives and confirms — the report proposes, it
* never merges. */
function reviewGroup(group: DuplicateGroup) {
reviewingGroup.value = group;
selectedIds.value = new Set(group.note_ids);
canonicalId.value = group.note_ids[0] ?? null;
showMergeModal.value = true;
}
function exitSelectMode() {
selectMode.value = false;
selectedIds.value = new Set();
reviewingGroup.value = null;
}
function toggleSelectMode() {
if (selectMode.value) exitSelectMode();
@@ -77,6 +124,15 @@ function openMerge() {
canonicalId.value = selectedList.value[0]?.id ?? null;
showMergeModal.value = true;
}
/** Dismiss the modal. Clears the reviewed group too — leaving it set would keep
* selectedList pinned to a corpus-wide set the operator has walked away from. */
function closeMerge() {
showMergeModal.value = false;
if (reviewingGroup.value) {
reviewingGroup.value = null;
selectedIds.value = new Set();
}
}
async function doMerge() {
const target = canonicalId.value;
if (target == null) return;
@@ -87,8 +143,13 @@ async function doMerge() {
await mergeSnippets(target, sources);
toast.show(`Merged ${sources.length} snippet${sources.length > 1 ? "s" : ""} in`);
showMergeModal.value = false;
const wasSuggested = reviewingGroup.value !== null;
exitSelectMode();
await loadSnippets();
// The merged-away records are gone, so a stale report would keep offering
// them. Re-run it rather than clearing, so the operator can work through
// several groups without re-triggering the scan each time.
if (wasSuggested && dupChecked.value) await loadDuplicates();
} catch {
toast.show("Failed to merge snippets", "error");
} finally {
@@ -259,6 +320,38 @@ function usageTitle(s: SnippetListItem): string {
>
{{ needsAttentionOnly ? "Needs attention · filtering" : "Needs attention" }}
</button>
<button
class="btn-ghost"
:disabled="dupLoading"
title="Look for snippets already recorded that resemble each other closely enough to be worth merging"
@click="loadDuplicates"
>
{{ dupLoading ? "Checking…" : "Find duplicates" }}
</button>
</div>
<!-- Near-duplicate report. Only ever a proposal — merging is a separate,
confirmed act, and the operator chooses which record survives. -->
<div v-if="dupChecked && !dupLoading" class="dup-panel">
<p v-if="!duplicateGroups.length" class="dup-empty">
No near-duplicates found. Nothing recorded resembles anything else closely
enough to be worth merging.
</p>
<template v-else>
<p class="dup-head">
{{ duplicateGroups.length }} possible duplicate{{ duplicateGroups.length > 1 ? " sets" : " set" }}
— review each before merging; a set is a suggestion, not a verdict.
</p>
<div v-for="(g, i) in duplicateGroups" :key="i" class="dup-group">
<div class="dup-members">
<span v-for="s in g.snippets" :key="s.id" class="dup-member">
{{ splitTitle(s.title).name }}
</span>
</div>
<span class="dup-score">{{ Math.round(g.top_score * 100) }}% alike</span>
<button class="btn-ghost dup-action" @click="reviewGroup(g)">Review &amp; merge</button>
</div>
</template>
</div>
<!-- Reverse lookup: what's already kept in this repo / file / symbol. -->
@@ -389,7 +482,7 @@ function usageTitle(s: SnippetListItem): string {
<!-- Merge modal -->
<teleport to="body">
<div v-if="showMergeModal" class="modal-overlay" @click.self="showMergeModal = false">
<div v-if="showMergeModal" class="modal-overlay" @click.self="closeMerge">
<div class="modal-card" role="dialog" aria-modal="true" aria-label="Merge snippets">
<h3 class="modal-title">Merge snippets</h3>
<p class="modal-desc">
@@ -409,7 +502,7 @@ function usageTitle(s: SnippetListItem): string {
</label>
</div>
<div class="modal-actions">
<button class="modal-btn" @click="showMergeModal = false">Cancel</button>
<button class="modal-btn" @click="closeMerge">Cancel</button>
<button
class="modal-btn modal-btn-primary"
:disabled="merging || canonicalId == null"
@@ -686,6 +779,63 @@ function usageTitle(s: SnippetListItem): string {
color: var(--color-text-muted);
}
/* Near-duplicate report */
.dup-panel {
margin-bottom: 1.25rem;
padding: 0.85rem 1rem;
border: 1px solid var(--color-border);
border-radius: 8px;
background: var(--color-surface-alt, var(--color-surface));
}
.dup-empty,
.dup-head {
margin: 0 0 0.5rem;
font-size: 0.85rem;
color: var(--color-text-muted);
}
.dup-empty {
margin-bottom: 0;
}
.dup-group {
display: flex;
align-items: center;
gap: 0.75rem;
flex-wrap: wrap;
padding: 0.5rem 0;
border-top: 1px solid var(--color-border);
}
.dup-members {
display: flex;
gap: 0.4rem;
flex-wrap: wrap;
flex: 1 1 20rem;
min-width: 0;
}
.dup-member {
font-size: 0.8rem;
padding: 0.1rem 0.45rem;
border-radius: 4px;
background: color-mix(in srgb, var(--color-text-muted) 12%, transparent);
/* Long snippet names must not push the row into a horizontal scroll. */
overflow-wrap: anywhere;
}
.dup-score {
font-size: 0.75rem;
color: var(--color-text-muted);
font-variant-numeric: tabular-nums;
white-space: nowrap;
}
.dup-action {
white-space: nowrap;
}
/* Drift is a stronger signal than dead weight: the record may be actively
misleading, not merely unused. Danger tone, and it sits first in the footer. */
.drift-tag {