feat(dedup): per-kind duplicate-report floors — notes/tasks default 0.93
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 19s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 53s
CI & Build / Build & push image (push) Successful in 42s

At chunk grain (#280) a note-pair's similarity is its closest chunk pair,
so the shared 0.82 floor saturated the note/task reports with related
families (38 note / 155 task groups against the 200-pair cap, measured
2026-08-09). Split kb_duplicate_threshold into per-kind settings keys
with per-kind defaults: snippet 0.82 (single-chunk, scale unchanged),
note/task 0.93 (points the report at genuinely-alike records). Settings
UI grows the two new knobs; report entrypoints inherit the change via
get_duplicate_threshold(user_id, kind).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-08-09 10:33:11 -04:00
co-authored by Claude Fable 5
parent 23cb396e65
commit 272b7dbddf
3 changed files with 135 additions and 38 deletions
+29 -11
View File
@@ -318,30 +318,45 @@ async def find_duplicate_note(
# scope here is set by what the operator can actually act on, not by what they
# can see.
#
# WHY A LOWER THRESHOLD THAN THE GATE. The gate BLOCKS a write at 0.90 and has to
# be unforgiving of noise. This report only makes a suggestion the operator
# reviews, so it can afford to be looser and catch the pairs the gate lets
# through — which are precisely the ones that accumulated. It is a setting rather
# WHY THE FLOOR IS PER-KIND. Chunked embeddings (#280) changed what a pair
# score MEANS for multi-chunk records: a note-pair's similarity is its closest
# chunk pair, so any family of related long records — a dev-log run, a research
# fan-out — clears a floor that whole-document vectors used to dilute below it.
# Measured on the live corpus (2026-08-09): at 0.82 the note/task reports
# saturate the pair cap with related-but-distinct families, while 0.93+ returns
# the genuinely-alike records. Snippets are single-chunk (short by nature), so
# their similarity scale never shifted and they keep the old floor.
#
# Snippets sit BELOW the 0.90 write gate — the report catches what the gate
# lets through. Notes/tasks sit ABOVE it, and that is not a contradiction: the
# gate compares a new record against best-matching chunks too, but it blocks a
# WRITE and must stay forgiving, while the report proposes a REVIEW and at
# chunk grain 0.90 would still drown it in families. Each is a setting rather
# than a constant (rule #25) because the right value depends on how uniform a
# corpus is, and nobody can guess that from here.
DUPLICATE_THRESHOLD_KEY = "kb_duplicate_threshold"
DUPLICATE_DEFAULT_THRESHOLD = 0.82
DUPLICATE_THRESHOLD_KEYS = {
"snippet": "kb_duplicate_threshold_snippet",
"note": "kb_duplicate_threshold_note",
"task": "kb_duplicate_threshold_task",
}
DUPLICATE_DEFAULT_THRESHOLDS = {"snippet": 0.82, "note": 0.93, "task": 0.93}
# Hard cap on returned pairs. A pathologically uniform corpus is O(n²) pairs, and
# a report nobody can read is not a report.
_MAX_DUPLICATE_PAIRS = 200
async def get_duplicate_threshold(user_id: int) -> float:
"""The user's near-duplicate similarity floor, clamped to [0, 1]."""
async def get_duplicate_threshold(user_id: int, kind: str = "snippet") -> float:
"""The user's near-duplicate similarity floor for `kind`, clamped to [0, 1]."""
from scribe.services.settings import get_setting
default = DUPLICATE_DEFAULT_THRESHOLDS[kind]
try:
value = float(await get_setting(
user_id, DUPLICATE_THRESHOLD_KEY, str(DUPLICATE_DEFAULT_THRESHOLD)
user_id, DUPLICATE_THRESHOLD_KEYS[kind], str(default)
))
except (TypeError, ValueError):
value = DUPLICATE_DEFAULT_THRESHOLD
value = default
return min(1.0, max(0.0, value))
@@ -503,7 +518,10 @@ async def find_duplicate_records(
"""
if kind not in _REPORT_KINDS:
raise ValueError(f"kind must be one of {_REPORT_KINDS}, not {kind!r}")
floor = await get_duplicate_threshold(user_id) if threshold is None else threshold
floor = (
await get_duplicate_threshold(user_id, kind)
if threshold is None else threshold
)
floor = min(1.0, max(0.0, floor))
max_distance = min(2.0, max(0.0, 1.0 - floor))