feat(snippets): near-duplicate finder — surface the sets worth merging
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 14s
CI & Build / integration (push) Successful in 36s
CI & Build / Python tests (push) Successful in 55s
CI & Build / Build & push image (push) Successful in 44s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 14s
CI & Build / integration (push) Successful in 36s
CI & Build / Python tests (push) Successful in 55s
CI & Build / Build & push image (push) Successful in 44s
#231's premise was unifying reusable things already scattered as one-offs. The create gate PREVENTS a new duplicate and merge_snippets CURES one you point it at, but nothing FOUND the duplicates already in the record — someone had to notice them by hand, which is the exact failure the Drafter exists to remove. One indexed self-join over note_embeddings, not an N² Python scan: pgvector's cosine distance is the same operator semantic search uses, so a similarity floor is a distance ceiling and the work stays in Postgres. `left.note_id < right.note_id` yields each unordered pair once and drops the self-pair that would otherwise dominate the ranking. Pairs are collapsed into merge SETS by connected components. Transitive on purpose: A~B plus B~C puts all three together even when A and C don't directly clear the bar, which is what merge actually does (it folds every source into one survivor). The cost is that a chain of mild resemblances can rope in a member that isn't really alike — so the UI presents a set as a proposal, shows the members, and never merges without a confirm. Two scope decisions worth naming: - OWN snippets only. merge_snippets requires one owner across the set, so surfacing someone else's would propose a merge that cannot be performed. The report is bounded by what the operator can act on, not what they can see. - Threshold defaults to 0.82, LOOSER than the write gate's 0.90, and is a setting rather than a constant (rule #25). The gate blocks a create and has to be unforgiving of noise; this only suggests a merge under review, so it must reach further or it would never surface the pairs the gate already let through — which are precisely the ones that accumulated. Fixes a real bug in the merge flow while wiring the UI: selectedList filtered the selection against the CURRENT PAGE, and doMerge derives its source ids from that list. A corpus-wide suggested group with off-page members would have rendered incomplete and silently merged only the visible subset. A group under review is now the authority for that list. Refs #2088 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -205,6 +205,24 @@ async def update_snippet_route(snippet_id: int):
|
||||
return jsonify(out)
|
||||
|
||||
|
||||
@snippets_bp.route("/duplicates", methods=["GET"])
|
||||
@login_required
|
||||
async def duplicate_snippets_route():
|
||||
"""Near-duplicate snippets already recorded, grouped into merge candidates.
|
||||
|
||||
Registered ABOVE the `/<int:snippet_id>` routes on purpose — Quart matches
|
||||
an int converter before a static segment either way, but keeping the literal
|
||||
path first makes the precedence obvious to the next person reading this."""
|
||||
uid = get_current_user_id()
|
||||
try:
|
||||
threshold = float(request.args.get("threshold", 0) or 0)
|
||||
except (TypeError, ValueError):
|
||||
threshold = 0.0
|
||||
return jsonify(await dedup_svc.find_duplicate_snippets(
|
||||
uid, threshold=threshold if threshold > 0 else None
|
||||
))
|
||||
|
||||
|
||||
@snippets_bp.route("/<int:snippet_id>/verify", methods=["POST"])
|
||||
@login_required
|
||||
async def verify_snippet_route(snippet_id: int):
|
||||
|
||||
Reference in New Issue
Block a user