feat(dedup): per-kind duplicate-report floors — notes/tasks default 0.93
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 19s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 53s
CI & Build / Build & push image (push) Successful in 42s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 19s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 53s
CI & Build / Build & push image (push) Successful in 42s
At chunk grain (#280) a note-pair's similarity is its closest chunk pair, so the shared 0.82 floor saturated the note/task reports with related families (38 note / 155 task groups against the 200-pair cap, measured 2026-08-09). Split kb_duplicate_threshold into per-kind settings keys with per-kind defaults: snippet 0.82 (single-chunk, scale unchanged), note/task 0.93 (points the report at genuinely-alike records). Settings UI grows the two new knobs; report entrypoints inherit the change via get_duplicate_threshold(user_id, kind). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -318,30 +318,45 @@ async def find_duplicate_note(
|
||||
# scope here is set by what the operator can actually act on, not by what they
|
||||
# can see.
|
||||
#
|
||||
# WHY A LOWER THRESHOLD THAN THE GATE. The gate BLOCKS a write at 0.90 and has to
|
||||
# be unforgiving of noise. This report only makes a suggestion the operator
|
||||
# reviews, so it can afford to be looser and catch the pairs the gate lets
|
||||
# through — which are precisely the ones that accumulated. It is a setting rather
|
||||
# WHY THE FLOOR IS PER-KIND. Chunked embeddings (#280) changed what a pair
|
||||
# score MEANS for multi-chunk records: a note-pair's similarity is its closest
|
||||
# chunk pair, so any family of related long records — a dev-log run, a research
|
||||
# fan-out — clears a floor that whole-document vectors used to dilute below it.
|
||||
# Measured on the live corpus (2026-08-09): at 0.82 the note/task reports
|
||||
# saturate the pair cap with related-but-distinct families, while 0.93+ returns
|
||||
# the genuinely-alike records. Snippets are single-chunk (short by nature), so
|
||||
# their similarity scale never shifted and they keep the old floor.
|
||||
#
|
||||
# Snippets sit BELOW the 0.90 write gate — the report catches what the gate
|
||||
# lets through. Notes/tasks sit ABOVE it, and that is not a contradiction: the
|
||||
# gate compares a new record against best-matching chunks too, but it blocks a
|
||||
# WRITE and must stay forgiving, while the report proposes a REVIEW and at
|
||||
# chunk grain 0.90 would still drown it in families. Each is a setting rather
|
||||
# than a constant (rule #25) because the right value depends on how uniform a
|
||||
# corpus is, and nobody can guess that from here.
|
||||
|
||||
DUPLICATE_THRESHOLD_KEY = "kb_duplicate_threshold"
|
||||
DUPLICATE_DEFAULT_THRESHOLD = 0.82
|
||||
DUPLICATE_THRESHOLD_KEYS = {
|
||||
"snippet": "kb_duplicate_threshold_snippet",
|
||||
"note": "kb_duplicate_threshold_note",
|
||||
"task": "kb_duplicate_threshold_task",
|
||||
}
|
||||
DUPLICATE_DEFAULT_THRESHOLDS = {"snippet": 0.82, "note": 0.93, "task": 0.93}
|
||||
# Hard cap on returned pairs. A pathologically uniform corpus is O(n²) pairs, and
|
||||
# a report nobody can read is not a report.
|
||||
_MAX_DUPLICATE_PAIRS = 200
|
||||
|
||||
|
||||
async def get_duplicate_threshold(user_id: int) -> float:
|
||||
"""The user's near-duplicate similarity floor, clamped to [0, 1]."""
|
||||
async def get_duplicate_threshold(user_id: int, kind: str = "snippet") -> float:
|
||||
"""The user's near-duplicate similarity floor for `kind`, clamped to [0, 1]."""
|
||||
from scribe.services.settings import get_setting
|
||||
|
||||
default = DUPLICATE_DEFAULT_THRESHOLDS[kind]
|
||||
try:
|
||||
value = float(await get_setting(
|
||||
user_id, DUPLICATE_THRESHOLD_KEY, str(DUPLICATE_DEFAULT_THRESHOLD)
|
||||
user_id, DUPLICATE_THRESHOLD_KEYS[kind], str(default)
|
||||
))
|
||||
except (TypeError, ValueError):
|
||||
value = DUPLICATE_DEFAULT_THRESHOLD
|
||||
value = default
|
||||
return min(1.0, max(0.0, value))
|
||||
|
||||
|
||||
@@ -503,7 +518,10 @@ async def find_duplicate_records(
|
||||
"""
|
||||
if kind not in _REPORT_KINDS:
|
||||
raise ValueError(f"kind must be one of {_REPORT_KINDS}, not {kind!r}")
|
||||
floor = await get_duplicate_threshold(user_id) if threshold is None else threshold
|
||||
floor = (
|
||||
await get_duplicate_threshold(user_id, kind)
|
||||
if threshold is None else threshold
|
||||
)
|
||||
floor = min(1.0, max(0.0, floor))
|
||||
max_distance = min(2.0, max(0.0, 1.0 - floor))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user