5 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Fable 5	041d8defbc	feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4) ... A note's relevance is now its best chunk's similarity, everywhere: - semantic_search_notes keeps the indexed raw-distance top-k and over-fetches chunk rows (x4, composing with the x3 supersession over-fetch), then collapses to first-appearance-per-note — rows arrive distance-ordered, so first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject, write-path, gate) inherits through the one function. - list_notes semantic q swaps its join for a correlated MIN-distance subquery — the join would have repeated a long note once per matching chunk and made total count chunks. - the duplicate report groups its self-join by note pair on MIN(distance): pair similarity = closest chunk pair, and the < join now also drops cross-chunk self-pairs that would flag every long note against itself. - the write gate queries once per chunk of the candidate (capped at 8), so a note duplicating an existing record in ONE SECTION is caught — the whole-document query diluted exactly the section that mattered. Integration test now seeds a two-chunk note and pins the collapse against real pgvector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:51:01 -04:00
bvandeusen	d7039dc17c	feat(dedup): the duplicate report reaches notes and tasks, with per-kind cures ... Step 5 of #278, folding in #2534. The operator's no-gate decision for the web UI (#2482 — "an llm attached to this surface is the corrections system") has a precondition nobody had built: the corrector has to be able to SEE what needs correcting. find_duplicate_snippets had no equivalent for notes or tasks, so a duplicate note was only ever noticed by accident. find_duplicate_records(kind="snippet"|"note"|"task") — the same indexed self-join, parameterised. Tasks are notes with a status, not a note_type, so the kind split is a status predicate; mixing them would propose folding a to-do into a write-up. find_duplicate_snippets stays as a wrapper because both surfaces and SnippetListView consume it by name. What differs by kind is the CURE, and the report says so in a `suggestion` field rather than leaving the caller to guess: snippet merge — lossless, the survivor keeps every call site note NEVER merge. A correction pair → supersedes on the newer; state smeared across dated records → extract to the System's reference note; genuinely parallel → leave alone. Choosing needs the records READ, which is the agent's job — so non-snippet groups carry `members` with dates and any `existing_supersessions` already declared inside the group. A pair someone ruled on is not an open question. task usually the same work opened twice — keep the one with the history, cancel the other with a pointer. The snippet sibling filter stays snippet-only: it keys on symbol/code_sha, which other kinds don't carry — and for them a look-alike is a finding. Surfaces: MCP find_duplicate_records (classified into _READ_ONLY_TOOLS — the completeness test would have caught the omission), REST /api/notes/duplicates, and a KnowledgeView panel mirroring SnippetListView's — links only, no merge button, because for notes the report proposes and the correction is a read- and-decide act. The panel follows the type filter and clears when it changes, so a note report can't linger under a task view. Correcting the task's own premise: it claimed the snippet report had "no view consuming it" — stale; SnippetListView has consumed it since it shipped. The UI gap was only ever notes/tasks. Answers the question carried from #2482: yes, the update routes on BOTH surfaces can turn a record into a duplicate — the gate is create-time by design. This report is the mechanism that catches it after the fact, which is the model the operator chose. Refs #278, #2547	2026-08-08 18:51:49 -04:00
bvandeusen	24d071619b	fix(dedup): compare the artefact, not the prose describing it ... The snippet gate was reading the wrong field, and #2464's UI recipes made it measurable in both directions at once: .btn-danger vs .btn-danger-outline 0.92 siblings, BLOCKED .btn-primary re-recorded verbatim under a different name <0.90 a literal copy, ALLOWED The second is what settles it. Identical code at an identical repo·path·symbol sailed through because the description differed, while two deliberately parallel variants were refused because theirs did not. A snippet's embedded document is mostly prose ABOUT the code, so no threshold fixes this: lowering it blocks more siblings, raising it admits more copies. So structure decides. Two exact signals, both index-served off the notes.data mirror that already exists, no migration and no backfill: location the same named thing in the same file. Requires BOTH path and symbol — a path alone is a directory of artefacts, and matching on it would refuse every second recipe from one stylesheet. code byte-identical code anywhere, via the same fingerprint the drift check uses. The semantic arm survives as a backstop for a genuine reword that shares neither, raised to 0.96 so it sits above the 0.92 band where real variants live. Structural hits say what they matched instead of hedging with "similar", and point at merge rather than update — two records of one artefact is what merge exists to fold back together. find_duplicate_snippets gets the same correction: pairs where both snippets name a symbol, name DIFFERENT symbols, and hold different code are variants, not copies. Without it a design system's button family reports as one merge set — eight recipes, every direct pair over the floor, top score 0.92, one click from collapsing a component family. The cost is real and stated in the code: a helper recorded twice under two names no longer reports. That trade favours the report being usable, and same-symbol and unnamed duplicates — how re-recording usually looks — still surface. The filter fails open, so a lookup failure degrades to the old unfiltered report rather than to a reassuring empty one. resolve_locations extracted: compose_body, create_snippet and now the gate each had their own copy of the repo/path/symbol shorthand fallback, and the gate is the one where a disagreement would mean matching a location the record won't be stored with. Applied to both create surfaces (#33) — the web UI must not be the way to record what the agent was stopped from writing. Refs #2518, #2464	2026-08-06 11:31:02 -04:00
bvandeusenandClaude Opus 4.8	dd1fc2d506	feat(mcp): extend dedup gate to create_rule / create_project_rule ... Completes the Phase 5 follow-up: rules now get the same update-over-create gate. Title-based only (rules aren't a semantic-retrieval/RAG surface), scoped to the same topic (rulebook rule) or same project (project rule). force=true overrides; fail-open like the note/task gate. Deferred-item decisions (operator): REST/web gating SKIPPED (kept MCP-only — humans rarely double-create and a hard block needs UI affordance); orphan scope kept orphan↔orphan (no change). So this rule gate is the only remaining build. - services/dedup.py: find_duplicate_rule(title, topic_id|project_id). - create_rule + create_project_rule: force param + gate. - tests: rule title match, scope-required guard, tool gate (block + force). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:43:17 -04:00
bvandeusenandClaude Opus 4.8	322cbc3b5e	feat(mcp): Phase 5 — write-time near-duplicate gate (update-over-create) ... #755 Phase 5. create_note / create_task now BLOCK a near-duplicate instead of silently inserting: they return {"duplicate": true, "existing_id", message} pointing at the record to UPDATE. Fights store bloat and stale competing copies that semantic search (RAG) would otherwise resurface for reconciliation. A force=true override creates anyway for genuinely-distinct records. - services/dedup.py: find_duplicate_note — two signals, scoped to owner + same project + same kind: (1) normalized-title exact match (cheap, always); (2) semantic cosine ≥ 0.90 but ONLY when body ≥ 200 chars (short/title-only embeddings false-positive — the pre-pivot lesson). Project-less (orphan) records compare only to other orphans on BOTH signals (orphan_only on the semantic call) — they're not matched across every project. - Gate wired into the MCP create_note/create_task tools (the LLM write path) with force override; _INSTRUCTIONS documents the duplicate response + force. - Opt-in by design: the service helper is only called from the interactive create tools. Internal/programmatic creates (recurrence spawn, imports) go straight through services.create_note and are NOT gated — a recurring task spawning its next same-titled instance must not be blocked. - Scope v1: MCP tools only. REST/web (human CRUD, needs a UI affordance) and create_rule (not a RAG surface; _INSTRUCTIONS already steer it) are follow-ups. - tests: dedup service (title/semantic/body-gate/type-filter) + tool gate (blocks, force bypasses) for notes and tasks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:21:35 -04:00