13 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Fable 5	041d8defbc	feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4) ... A note's relevance is now its best chunk's similarity, everywhere: - semantic_search_notes keeps the indexed raw-distance top-k and over-fetches chunk rows (x4, composing with the x3 supersession over-fetch), then collapses to first-appearance-per-note — rows arrive distance-ordered, so first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject, write-path, gate) inherits through the one function. - list_notes semantic q swaps its join for a correlated MIN-distance subquery — the join would have repeated a long note once per matching chunk and made total count chunks. - the duplicate report groups its self-join by note pair on MIN(distance): pair similarity = closest chunk pair, and the < join now also drops cross-chunk self-pairs that would flag every long note against itself. - the write gate queries once per chunk of the candidate (capped at 8), so a note duplicating an existing record in ONE SECTION is caught — the whole-document query diluted exactly the section that mattered. Integration test now seeds a two-chunk note and pins the collapse against real pgvector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:51:01 -04:00
bvandeusenandClaude Fable 5	0e70a3896b	feat(embeddings): per-chunk rows — schema, write path, version-aware backfill (#280 steps 2+3) ... note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus chunk_text (what this vector actually encodes) and chunker_version. Migration 0077 clears the table — embeddings are derived (0067 precedent) and the old whole-document rows are indistinguishable from single-chunk notes, so the startup backfill regenerates the corpus at the new shape. The backfill is now version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds exactly the stale notes, not another wipe. upsert_note_embedding takes (title, body) and chunks internally — one path for the write path, the recurrence spawn and the backfill. The recurrence spawn's own embed call is deleted outright: create_note already embeds via embed_note (#2056), so the spawn was a second copy of the rule. An emptied record now CLEARS its stale vectors instead of leaving them findable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:47:34 -04:00
bvandeusenandClaude Fable 5	6b5043a69c	feat(embeddings): chunk_document — the chunked document shape (#280 step 1) ... bge-small reads 512 tokens and fastembed truncates silently, so a single whole-document vector permanently lost everything past ~400 words. The new shape: split at markdown headings (fence-aware), merge small sections, split oversize ones at paragraph boundaries, title-anchor every chunk, repeat the section heading on continuation pieces. A record that fits the window yields exactly one chunk identical to the historical title\nbody shape, so the corpus's sharpest records are byte-for-byte unaffected. CHUNKER_VERSION added so later shape changes re-embed by version comparison instead of a table wipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:43:51 -04:00
bvandeusen	f11a547cd2	fix(lists): shared-project records appear, and a list's q means what search means ... #2462, both decided halves. ## The ACL defect list_notes filtered on Note.user_id == user_id with no scope parameter at all — never a per-call decision, the capability was absent. query_knowledge beside it was deliberately browse-scoped with a comment saying why. So a task in a shared project was invisible in list_tasks, enter_project's open-task list, the SessionStart todo count and the web UI's task views, while the same project's notes and snippets appeared. Now the shared clause: notes_visibility_clause(user_id, "browse"). Browse and not read, per decision #2094 — an ambient list must never surface a record someone shared one-to-one; those stay search-only. This is the remaining half of a fix made twice before (#2159 widened fetch, #2092 widened meaning), and the guard below is what stops a fourth half appearing. Guarded by source inspection in test_retrieval_scopes: every list-shaped service references a shared visibility clause AND carries no bare Note.user_id comparison that would quietly re-narrow it. An owner-only list returns correct-looking rows and simply omits the shared ones — the shape no behavioural test catches. ## q is semantic (operator: "make it match") The UI's note list keyword-matched while the UI's Browse search semantic-matched, over the same records, with nothing saying so. Now one meaning: q joins the embedding index and orders by cosine distance at the interactive floor, with every lifecycle filter still applied in the same indexed query. Relevance ordering wins over `sort` when q is present — a query is a relevance claim, and sorting its results by date would shuffle the answer. ILIKE survives only as the embedder-down fallback: degraded, never empty. Stated position: superseded records are NOT demoted in list-q. The penalty reorders a top-k; reordering a paginated, counted list would make page boundaries lie. The search surfaces carry the demotion. ## The interactive floor becomes one constant INTERACTIVE_SEARCH_THRESHOLD = 0.3 in embeddings.py, consumed by routes/search.py (was a commented constant), knowledge.py (was a bare literal), and the new list-q. This closes #2463's finding 3 early — the same number lived in two files with the reasoning attached to only one. ## Deferred, deliberately The return-shape unification (ORM objects vs dicts) stays undone. It is a 14-caller refactor whose motivation — callers being unable to swap paths — shrinks now that both paths share the clause and the meaning of q. If swap pressure recurs it deserves its own change, not a rider on an ACL fix. Refs #2462, #2463	2026-08-08 19:58:01 -04:00
bvandeusen	3f1523b19f	feat(systems): read-side teeth — the vocabulary at session start, a search filter, and the state/chronicle instructions ... Step 4 of #278, product half. The audit that motivated it: one System in project 2, thirty records tagged, nothing since July 28 — three days after the feature landed. Not a discipline failure; retrieval was completely blind to the association (zero references in embeddings, knowledge, search, auto-inject, or enter_project), so tagging was a write-side label with no read-side payoff, and labels nobody reads don't get maintained. Three changes, ordered by what makes the others workable: 1. enter_project returns the project's Systems (id, name, first line of the charter). Load-bearing for the tagging instruction: you cannot ask an agent to check a record against a vocabulary it never sees. Trimmed because it rides on every session start; the full charter stays get_system's job. Present-and-empty rather than absent when a project has none — "no named areas yet" is information the create-the-System instruction acts on. 2. search accepts system_id, MCP and REST (#33). Implemented once in semantic_search_notes as an EXISTS against record_systems — an association filter deciding candidate-set membership before scoring, like project_id, not a ranking signal. The REST route's missing project filter stays #2463's: it carries a default-scope UI decision this change must not preempt. 3. The instructions (#119, _INSTRUCTIONS + using-scribe skill; plugin 0.1.25 for the cache): - Tag as you write, with an executable test — "would someone investigating that subsystem want this in the pile list_system_records returns?" — rather than "tag appropriately", which is what died. - Create the System when the area has no record: the two-or-more test snippets use, plus "don't wait to be asked to name an area that plainly exists", because the agent's default was leaving un-modelled areas un-modelled forever. - State vs chronicle: dev-logs are written once and never rewritten; durable findings live in the System's reference note, updated in place — safe because note versions are the changelog, which has existed since the feature shipped and was never named as one. list_system_records' docstring now sells it as the way to READ a subsystem, reference note first. No auto-inject boost by System — vocabulary and filter first, measure before adding ranking behaviour (the #2486 lesson). Refs #278, #2546	2026-08-08 18:19:58 -04:00
bvandeusen	f20c019f2a	feat(supersession): demote what a later note overtook, and label it ... Step 3 of #278. First step with visible effect. ## Where the demotion happens, and why not in SQL Applied AFTER the pgvector fetch, over an over-fetched candidate set, not as part of the ORDER BY. Ordering by `distance + penalty` would be exact and would turn an indexed top-k into a scan-and-sort of every embedded note — the HNSW index from migration 0067 can only serve a raw-distance ordering. So the query fetches 3x the requested rows by raw distance and the re-rank happens in Python. Demoting after a LIMIT k with no over-fetch would have been theatre: the cut already happened, so a superseded record pushed down still sits in the results and the live record that should have replaced it was never fetched. The cost is stated in the code: a live record outside the over-fetch window cannot be promoted in. With a 0.05 penalty against neighbours ~0.014 apart, that needs the true answer more than three ranks down, which no observed query approaches. ## Demote, never hide — enforced in three places The penalty applies to the RANKING score, not to the relevance threshold. The floor decides whether a record is relevant at all; the penalty decides which relevant record comes first. Applying it to the floor would drop a superseded record out of the results entirely, which is the one thing this must not do. It is small on purpose. Supersession is a claim about SOME of a record's content, so one that strongly answers a question nothing else answers still surfaces — just behind anything comparable that is current. Its test asserts both bounds, the upper one citing the operator's constraint rather than an optimisation. And every test here that could be satisfied by dropping a record instead asserts the record is still present. ## The dedup gate opts out A superseded record is still a duplicate of what you are about to write — the claim is that it is no longer current, not that it is gone. Demoting it there would let the same note be recorded a second time, and the second copy would be the one nothing warns about. ## The label Auto-inject marks a superseded line SUPERSEDED with a pointer to check the later record. One query for the whole menu. An agent handed stale material with nothing marking it acts on it with full confidence, which is worse than never having surfaced it — the ranking is only half the fix. Fails open: a supersession lookup error returns unpenalised results rather than none, because unpenalised ranking is the behaviour that shipped for months and a broken search is not. Refs #278	2026-08-08 02:00:06 -04:00
bvandeusen	bbba0b3ae3	refactor(embeddings): one definition of the document a record is embedded as ... `f"{title}\n{body}"` was written out four times. #2486 found three — the write path, the recurring-task spawn, the startup backfill. The guard added here found the fourth immediately, and it was the one that mattered most. `dedup.find_duplicate_note` built the same string as a QUERY, compared against embedded documents. Shaped differently from the corpus it searches, the gate degrades silently: it still returns neighbours, just less apt ones, and nothing says the query and the index stopped agreeing. The spawn path has the same shape of risk — a recurring task embedded differently from everything else is ranked against documents it doesn't match. None of the four had diverged. That is what makes this worth doing now rather than after: they are identical today, so collapsing them is a no-op, and the whole point is that the next change to the shape can't hit three of four. Which is imminent. #2486 measured a dev-log separating from five unrelated dev-logs by 0.023 where a snippet separates by 0.153 — the difference being that a snippet states its purpose twice in a short document. Whether that shape is right is the open question; testing an alternative against four copies would mean testing a shape that isn't the one in production. This is the precondition the issue named. The guard is source inspection, matching the f-string pattern rather than a variable name, so a copy that renames its locals is still caught. A behavioural test cannot see this: an inlined copy produces the same string today and diverges the day the shape changes. Refs #2486	2026-08-07 13:01:45 -04:00
bvandeusenandClaude Opus 5	f8522fb28f	fix(retrieval): give reuse a slot, and let experience reach the write path ... Two mirror-image scoping mistakes, neither deliberate (#2246). AUTO-INJECT let every kind compete on raw cosine. That is fatal rather than merely imperfect here, because Scribe's project records are ABOUT software work: a task titled "surface snippets before the agent writes code" is a near-perfect lexical match for "write a function…" while answering none of it. Measured live, a prompt asking for a helper returned three records about BUILDING the retrieval system and zero snippets. Snippets are ~0.5% of the corpus, and the ratio worsens as the project record grows — which is the direction Scribe is meant to grow, so no threshold tuning fixes it. Now the best snippet or process takes the LAST slot when none won on score. Deliberately NOT held to the margin band: that band measures distance from the top overall score, and the top score is the very thing snippets lose to. It still must clear the configured threshold, so a weak snippet cannot buy the slot — silence stays the default. Skipped entirely when reuse already won, so the fix is invisible in the case it isn't needed. WRITE-PATH was snippets-only — the same mistake inverted. An issue recording "we tried this and it deadlocked" could never reach the moment that code was about to be written, though it is arguably the better prior art: it says what NOT to do. Widened to snippets plus recorded experience. That needed a filter the search layer couldn't express. "Experience" is issues plus dev-logs, which differ on is_task, so neither note_type nor is_task alone covers it. semantic_search_notes gains task_kind, which restricts TASKS to the given kinds while leaving non-task notes untouched — so note_type=("snippet", "note") + task_kind="issue" yields snippets, fixed problems and durable notes, without the open to-do list. note_type now accepts a sequence too. Non-snippet hits are labelled with their kind, because an unlabelled issue on that menu reads as "here is code to reuse", the opposite of what it says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-07-31 22:56:31 -04:00
bvandeusen	e0328f2b1c	feat(plugin): write-path trigger — offer prior art before code is rewritten ... The milestone headline. Auto-inject fires on the operator's prompt; the moment reuse is actually lost is later, when the agent decides mid-task to write a helper. A PreToolUse hook on Write|Edit now fires there. Channel: `additionalContext` with NO permissionDecision, so the note reaches Claude beside the tool result and the write is never blocked — a recall aid must not be able to stop the operator's work. Plain stdout would have been invisible to the model, and deny/ask would have made a nudge into a gate. Two arms, different in kind: - BY PLACE — a snippet recorded at this path (or its directory) is prior art by definition, not resemblance, so it is neither scored nor thresholded. This is what #2083's reverse lookup was built to answer. - BY MEANING — semantic search restricted to snippets (new `note_type` filter on semantic_search_notes) over the code about to be written. Place ranks first; the top-k cap spans both arms. Gates carried over from milestone 93 verbatim: threshold, margin, session dedup, titles-never-bodies. Own `source='write_path'` in retrieval_logs so precision is tunable separately — the docstring records that the place arm is unlogged and hands that to #2085. Its own on/off in Settings but the SAME threshold/top-k: one "how loud may Scribe be" knob is easier to reason about than two that drift, and splitting them later is then a data-backed change rather than a guess. Details worth keeping: the hook sends a REPO-RELATIVE path because that is how locations are recorded; the git remote resolves to a project and is never used as the location `repo` filter (different namespaces, would silently match nothing); the endpoint stays a GET because a read-scoped API key cannot POST and every other hook depends on that. plugin.json 0.1.17 -> 0.1.18. Refs #2082, milestone #232.	2026-07-28 09:08:17 -04:00
bvandeusenandClaude Opus 5	ef1dbdfc86	feat(acl): unify the retrieval scopes; mark shared rows in Knowledge browse ... Closes #2092 and the Knowledge-browse provenance gap. The two halves of a hybrid search disagreed: the keyword half honoured shares while the semantic half was pinned to NoteEmbedding.user_id, so a shared record was findable by wording and invisible by meaning — the case a semantic search exists to serve. semantic_search_notes now scopes on Note via a `scope` parameter, and each of its five callers declares which kind of act it is: mcp/tools/search.py read the agent asked routes/search.py read the user typed it knowledge.py (semantic) read matches the keyword half beside it plugin_context.py browse nobody asked; never a one-to-one share dedup.py own a verdict that blocks a write must not hinge on another person's notes That last one is the reason this isn't a single global widening: the dedup gate returns "update the existing one instead", so matching a stranger's record would refuse a legitimate create and point at something the caller can't edit. Scope defaults to "own" so a caller that forgets is wrong in the safe direction, and an unknown scope raises rather than falling back — a typo there would be a data-exposure bug. Auto-inject keeps the browse scope, which still admits a collaborator's note via a shared project. Its menu line is the only provenance an agent sees, so a foreign hit now reads: #12 "Title" (0.71) - shared by alex, treat as a suggestion. MCP and REST search results carry shared/owner too. Knowledge browse: the feed hydrates cards from /api/knowledge/batch rather than the list route, so both paths label rows now, and KnowledgeView shows "by <owner>" on records the viewer doesn't own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt	2026-07-25 23:06:52 -04:00
bvandeusenandClaude Opus 4.8	513019786e	feat(search): pgvector substrate — vector(384) + HNSW for semantic search ... Move semantic_search_notes off the full-table Python cosine scan onto a native pgvector column: indexed ORDER BY embedding <=> :q LIMIT k (HNSW, cosine). Migration 0067 enables the extension, converts the JSONB embedding column to vector(384) (stale-dim rows dropped and regenerated by the startup backfill), and builds the HNSW cosine index. Postgres image moves postgres:16-alpine -> pgvector/pgvector:pg17 across prod, quickstart, and CI. Scribe: project 2, milestone 93, task 1031. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xz4j1H7pjYSjKsEpgcNH5E	2026-06-22 20:10:15 -04:00
bvandeusenandClaude Opus 4.8	70ab3f38c6	chore: remove pre-pivot dead code + finish Scribe rebrand (#599 t1-3) ... - Header wordmark Fabled -> Scribe; fable:calendar-changed event -> scribe:calendar-changed; SettingsView CSS comment. - Drop dead Project.auto_summary + summary_updated_at columns (migration 0063) -- the Ollama-era summarizer is gone; model + 2 frontend types + projects test updated. - Remove pivot vestiges: diagnostics _curator_busy()/curator_busy heartbeat field, tz BRIEFING_DAY_START_HOUR/user_briefing_date dead aliases, the ignored 'model' param on get_embedding (+ its test). ruff src/ clean; CI is the gate. Part of scribe plan #599. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 16:16:44 -04:00
bvandeusenandClaude Opus 4.8	b255a0f90e	refactor: rename package fabledassistant -> scribe (code-only) ... Renames src/fabledassistant -> src/scribe and all imports, plus the default DB name and DB user/password (fabled -> scribe) in config + compose. 952 refs / 154 files. Reverses the old 'internal name stays fabledassistant' convention. Code-only: live databases are still physically named 'fabledassistant'. Deployed environments must set POSTGRES_DB / POSTGRES_USER (or rename the DB) since the defaults now resolve to 'scribe'. Repo (FabledScribe), git host (fabledsword), MCP (fabled-git) and the image name (fabledscribe) are intentionally unchanged. ruff check src/ clean locally; CI (typecheck + pytest) is the gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 15:48:35 -04:00