feat(snippets): near-duplicate finder — surface the sets worth merging
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 14s
CI & Build / integration (push) Successful in 36s
CI & Build / Python tests (push) Successful in 55s
CI & Build / Build & push image (push) Successful in 44s

#231's premise was unifying reusable things already scattered as one-offs.
The create gate PREVENTS a new duplicate and merge_snippets CURES one you
point it at, but nothing FOUND the duplicates already in the record —
someone had to notice them by hand, which is the exact failure the Drafter
exists to remove.

One indexed self-join over note_embeddings, not an N² Python scan:
pgvector's cosine distance is the same operator semantic search uses, so a
similarity floor is a distance ceiling and the work stays in Postgres.
`left.note_id < right.note_id` yields each unordered pair once and drops
the self-pair that would otherwise dominate the ranking.

Pairs are collapsed into merge SETS by connected components. Transitive on
purpose: A~B plus B~C puts all three together even when A and C don't
directly clear the bar, which is what merge actually does (it folds every
source into one survivor). The cost is that a chain of mild resemblances
can rope in a member that isn't really alike — so the UI presents a set as
a proposal, shows the members, and never merges without a confirm.

Two scope decisions worth naming:

- OWN snippets only. merge_snippets requires one owner across the set, so
  surfacing someone else's would propose a merge that cannot be performed.
  The report is bounded by what the operator can act on, not what they can
  see.

- Threshold defaults to 0.82, LOOSER than the write gate's 0.90, and is a
  setting rather than a constant (rule #25). The gate blocks a create and
  has to be unforgiving of noise; this only suggests a merge under review,
  so it must reach further or it would never surface the pairs the gate
  already let through — which are precisely the ones that accumulated.

Fixes a real bug in the merge flow while wiring the UI: selectedList
filtered the selection against the CURRENT PAGE, and doMerge derives its
source ids from that list. A corpus-wide suggested group with off-page
members would have rendered incomplete and silently merged only the
visible subset. A group under review is now the authority for that list.

Refs #2088

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-28 18:32:40 -04:00
co-authored by Claude Opus 5
parent 84c5c0dc81
commit 6db791965f
10 changed files with 547 additions and 8 deletions
+32
View File
@@ -23,6 +23,10 @@ const kbInjectEnabled = ref(true);
const kbInjectThreshold = ref("0.55");
const kbInjectTopK = ref("3");
const kbWritePathEnabled = ref(true);
// Near-duplicate report floor. Deliberately looser than the 0.90 write-time
// gate: that one BLOCKS a create and must be unforgiving of noise, this one only
// suggests a merge the operator reviews (services/dedup.py).
const kbDuplicateThreshold = ref("0.82");
const savingKbInject = ref(false);
const kbInjectSaved = ref(false);
@@ -68,8 +72,13 @@ async function saveRetention() {
async function saveKbInject() {
const t = Math.min(1, Math.max(0, Number(kbInjectThreshold.value) || 0));
const k = Math.min(10, Math.max(1, Math.floor(Number(kbInjectTopK.value) || 1)));
// `|| 0.82` not `|| 0`: an unparseable value here should fall back to the
// default, not to 0 — a 0 floor would report every snippet as a duplicate of
// every other one.
const dupT = Math.min(1, Math.max(0, Number(kbDuplicateThreshold.value) || 0.82));
kbInjectThreshold.value = String(t);
kbInjectTopK.value = String(k);
kbDuplicateThreshold.value = String(dupT);
savingKbInject.value = true;
kbInjectSaved.value = false;
try {
@@ -80,6 +89,7 @@ async function saveKbInject() {
// Its own switch, but deliberately the same threshold/ceiling — see
// WRITEPATH_ENABLED_KEY in services/plugin_context.py.
kb_writepath_enabled: kbWritePathEnabled.value ? 'true' : 'false',
kb_duplicate_threshold: String(dupT),
});
kbInjectSaved.value = true;
setTimeout(() => (kbInjectSaved.value = false), 2000);
@@ -464,6 +474,9 @@ onMounted(async () => {
kbInjectTopK.value = allSettings.kb_autoinject_top_k;
}
kbWritePathEnabled.value = allSettings.kb_writepath_enabled !== "false";
if (allSettings.kb_duplicate_threshold !== undefined) {
kbDuplicateThreshold.value = allSettings.kb_duplicate_threshold;
}
if (allSettings.notify_task_reminders !== undefined) {
notifyTaskReminders.value = allSettings.notify_task_reminders !== "false";
}
@@ -1211,6 +1224,25 @@ function formatUserDate(iso: string): string {
edit. Off = prior art surfaces only on your own prompts.
</p>
</div>
<div class="field">
<label for="kb-duplicate-threshold">Near-duplicate report threshold</label>
<input
id="kb-duplicate-threshold"
v-model="kbDuplicateThreshold"
type="number"
min="0"
max="1"
step="0.01"
class="input"
style="max-width: 8rem"
/>
<p class="field-hint">
How alike two snippets must be before the Snippets page suggests merging
them. Lower = more suggestions, more false pairs. Looser than the 0.90
used to block a duplicate at creation, because this only proposes a merge
you review — it never acts on its own.
</p>
</div>
<div class="actions">
<button class="btn-save" @click="saveKbInject" :disabled="savingKbInject">
{{ savingKbInject ? 'Saving' : 'Save' }}