12 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Fable 5	041d8defbc	feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4) ... A note's relevance is now its best chunk's similarity, everywhere: - semantic_search_notes keeps the indexed raw-distance top-k and over-fetches chunk rows (x4, composing with the x3 supersession over-fetch), then collapses to first-appearance-per-note — rows arrive distance-ordered, so first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject, write-path, gate) inherits through the one function. - list_notes semantic q swaps its join for a correlated MIN-distance subquery — the join would have repeated a long note once per matching chunk and made total count chunks. - the duplicate report groups its self-join by note pair on MIN(distance): pair similarity = closest chunk pair, and the < join now also drops cross-chunk self-pairs that would flag every long note against itself. - the write gate queries once per chunk of the candidate (capped at 8), so a note duplicating an existing record in ONE SECTION is caught — the whole-document query diluted exactly the section that mattered. Integration test now seeds a two-chunk note and pins the collapse against real pgvector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:51:01 -04:00
bvandeusenandClaude Fable 5	0e70a3896b	feat(embeddings): per-chunk rows — schema, write path, version-aware backfill (#280 steps 2+3) ... note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus chunk_text (what this vector actually encodes) and chunker_version. Migration 0077 clears the table — embeddings are derived (0067 precedent) and the old whole-document rows are indistinguishable from single-chunk notes, so the startup backfill regenerates the corpus at the new shape. The backfill is now version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds exactly the stale notes, not another wipe. upsert_note_embedding takes (title, body) and chunks internally — one path for the write path, the recurrence spawn and the backfill. The recurrence spawn's own embed call is deleted outright: create_note already embeds via embed_note (#2056), so the spawn was a second copy of the rule. An emptied record now CLEARS its stale vectors instead of leaving them findable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-08-08 23:47:34 -04:00
bvandeusen	f11a547cd2	fix(lists): shared-project records appear, and a list's q means what search means ... #2462, both decided halves. ## The ACL defect list_notes filtered on Note.user_id == user_id with no scope parameter at all — never a per-call decision, the capability was absent. query_knowledge beside it was deliberately browse-scoped with a comment saying why. So a task in a shared project was invisible in list_tasks, enter_project's open-task list, the SessionStart todo count and the web UI's task views, while the same project's notes and snippets appeared. Now the shared clause: notes_visibility_clause(user_id, "browse"). Browse and not read, per decision #2094 — an ambient list must never surface a record someone shared one-to-one; those stay search-only. This is the remaining half of a fix made twice before (#2159 widened fetch, #2092 widened meaning), and the guard below is what stops a fourth half appearing. Guarded by source inspection in test_retrieval_scopes: every list-shaped service references a shared visibility clause AND carries no bare Note.user_id comparison that would quietly re-narrow it. An owner-only list returns correct-looking rows and simply omits the shared ones — the shape no behavioural test catches. ## q is semantic (operator: "make it match") The UI's note list keyword-matched while the UI's Browse search semantic-matched, over the same records, with nothing saying so. Now one meaning: q joins the embedding index and orders by cosine distance at the interactive floor, with every lifecycle filter still applied in the same indexed query. Relevance ordering wins over `sort` when q is present — a query is a relevance claim, and sorting its results by date would shuffle the answer. ILIKE survives only as the embedder-down fallback: degraded, never empty. Stated position: superseded records are NOT demoted in list-q. The penalty reorders a top-k; reordering a paginated, counted list would make page boundaries lie. The search surfaces carry the demotion. ## The interactive floor becomes one constant INTERACTIVE_SEARCH_THRESHOLD = 0.3 in embeddings.py, consumed by routes/search.py (was a commented constant), knowledge.py (was a bare literal), and the new list-q. This closes #2463's finding 3 early — the same number lived in two files with the reasoning attached to only one. ## Deferred, deliberately The return-shape unification (ORM objects vs dicts) stays undone. It is a 14-caller refactor whose motivation — callers being unable to swap paths — shrinks now that both paths share the clause and the meaning of q. If swap pressure recurs it deserves its own change, not a rider on an ACL fix. Refs #2462, #2463	2026-08-08 19:58:01 -04:00
bvandeusen	45c6b1c88a	feat(supersession): the relation, and the dead column that stood where it should ... Step 1 of #278. Structure only — nothing reads or writes the new table yet. Old records outrank newer ones on the same subject because a similarity score cannot tell time. A note that accurately described how something worked in June is still accurate ABOUT June; it is just no longer the answer. Nothing recorded that, so nothing could act on it. `note_supersessions(superseder_id, superseded_id)`. The claim points FORWARD — the newer record names what it overtakes — because the older one cannot know it has been overtaken; asking it to record its own obsolescence is asking it to predict the future. A table rather than a column because the relation is genuinely many-to-many and partial, and both directions are hot: superseded_id answers "has this been overtaken?" at ranking time, superseder_id answers "what does this replace?" in a record view. An array column serves one and not the other. CASCADE is safe because trashing is not a delete — trash_svc stamps deleted_at, so a trashed note keeps its claims and restore brings them back. It fires only on purge_trash, where a claim about the row would be unactionable anyway. A CHECK rejects self-supersession, which under flat demotion would let a record demote itself. ## consolidated_at, and what it actually was Dropped. Written by nothing while serialised into every note and task payload as null — and worse, it implied a capability. The survey (#2483) read it as note consolidation modelled and abandoned. That was wrong, and the frontend is what says so: `TaskViewerView` rendered "✦ Auto-summarized from work logs" gated on this column. It is a survivor of the pre-pivot auto-summary subsystem (migration 0030), whose own column #599 removed. Not an unbuilt feature — an outlived one. So four more remnants went with it: the banner, its CSS, a `consolidatedAt` ref in TaskEditorView assigned and never read, and `.auto-summary-banner-editor` styling with zero template usage. That last one is presence-without-reference in the same family as the column itself. Dropped rather than repurposed for supersession, and the distinction is the point: consolidation folds records into one survivor and destroys the originals. Supersession is the opposite — both survive, the older ranks behind. Smuggling one in under a column named for the other would bury that in schema. ## The hard delete_note Removed, with a comment where it stood. Zero callers, and the danger was never that it ran — it is that it was findable by name. Someone wanting to delete a note greps `delete_note`, finds a function in the notes service with exactly the right signature, and permanently destroys a record every path downstream expects to be recoverable. The MCP tool of the same name already went through trash_svc; only the service function was the trap. Refs #278, #2483	2026-08-07 21:40:51 -04:00
bvandeusen	bbba0b3ae3	refactor(embeddings): one definition of the document a record is embedded as ... `f"{title}\n{body}"` was written out four times. #2486 found three — the write path, the recurring-task spawn, the startup backfill. The guard added here found the fourth immediately, and it was the one that mattered most. `dedup.find_duplicate_note` built the same string as a QUERY, compared against embedded documents. Shaped differently from the corpus it searches, the gate degrades silently: it still returns neighbours, just less apt ones, and nothing says the query and the index stopped agreeing. The spawn path has the same shape of risk — a recurring task embedded differently from everything else is ranked against documents it doesn't match. None of the four had diverged. That is what makes this worth doing now rather than after: they are identical today, so collapsing them is a no-op, and the whole point is that the next change to the shape can't hit three of four. Which is imminent. #2486 measured a dev-log separating from five unrelated dev-logs by 0.023 where a snippet separates by 0.153 — the difference being that a snippet states its purpose twice in a short document. Whether that shape is right is the open question; testing an alternative against four copies would mean testing a shape that isn't the one in production. This is the precondition the issue named. The guard is source inspection, matching the f-string pattern rather than a variable name, so a copy that renames its locals is still caught. A behavioural test cannot see this: an inlined copy produces the same string today and diverges the day the shape changes. Refs #2486	2026-08-07 13:01:45 -04:00
bvandeusenandClaude Opus 5	5c51e29f26	fix(embeddings): embed in the service, so every caller gets it (#2056) ... A note or task created through MCP was not semantically searchable until the next restart's backfill ran. Embedding fired at the five REST route handlers and nowhere else; the MCP tools call the service directly, so they skipped it. The shape of this bug is the reason to care: it is invisible on an instance that redeploys constantly (this one does, per rule 46) and permanent on one that doesn't. Rule 115 — the product has to stand up for the install that restarts twice a year, not just for the one that restarts hourly. Moved to services/notes.embed_note(), called from create_note and update_note, and deleted from all five routes. Every caller — REST, MCP, recurrence, snippets — now gets it by construction rather than by remembering. Two things fall out of having one implementation instead of six: - It uses note.user_id, the OWNER. The routes were inconsistent: some passed the caller's uid, some the owner's. On a shared record the caller's id mints a second embedding row that nothing reads. - services/snippets.py's _embed_snippet existed only because snippets are created via MCP and the routes couldn't cover them. Every one of its four call sites goes through notes_svc, so the helper and its four calls are gone, along with the eight test patches that existed to neutralise it. RuntimeError (no running loop — unit tests, scripts) and any indexing failure are both swallowed: a write that succeeded must not be failed by its index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs	2026-07-31 22:46:57 -04:00
bvandeusenandClaude Opus 5	cca40affe4	feat(snippets): add notes.data JSONB — the indexed mirror of snippet fields ... Milestone #232 step 1 (task #2081). Takes the enabler first rather than the write-path trigger: reverse lookup, drift checks and the duplicate finder all need to QUERY structured fields, and building them on body-regex first means writing them twice. #227 deferred this bag "unless body-convention ergonomics prove insufficient" — answering "which snippets live in this file?" by scanning every snippet and regexing its body is that condition being met. Migration 0070 adds `notes.data` (nullable JSONB) + a GIN index. The body is UNCHANGED and still what gets embedded and read by humans; `data` mirrors the same facts in a shape Postgres can index. Code is deliberately not copied into it — the body holds it, and duplicating a blob into the column we index around would be waste. - compose_data() builds the mirror, omitting empties so the column stays sparse - snippet_fields() prefers `data`, falling back to parsing the body. Rows written before 0070 have no `data` and are never backfilled, so a hand-edited body stays authoritative for them with no conversion deadline - create / update / merge all write body and mirror from the same merged field set, so the two can't drift; merge in particular has to grow the mirror with the survivor's location set or a merged snippet would be unfindable at the very call sites the merge just recorded Named `data`, not `metadata`, because that collides with SQLAlchemy's declarative Base.metadata — which is why the pre-0069 model had to map an awkward `entity_metadata` attribute. Not a revival of the column 0069 dropped: different name, different purpose, nothing reads the old shape. Two test fakes needed an explicit `data = None`: snippet_fields prefers `data` when truthy and an auto-MagicMock attribute is truthy, so every parsed field would have come back a MagicMock. Checked every fake reaching snippet code this time rather than waiting for CI (note 2109). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt	2026-07-26 18:25:55 -04:00
bvandeusenandClaude Opus 5	8b069cc93f	fix(acl): make the visibility predicates pure builders; repair CI ... Run 2888 failed with 7 tests down, both causes mine. The real problem was a design flaw, not the tests: readable_notes_clause and browsable_notes_clause each opened their own DB session to fetch the caller's group ids. That made them unmockable at the call site, so every unrelated service test suddenly had to know they existed and stub them — four modules broke the moment a service started calling one, and one of my own stubs patched the wrong name (readable_* where the code had moved to browsable_*). Fixed at the root: group membership is now a SUBQUERY rather than a fetched list, so both clauses are synchronous pure functions with no session. One fewer round-trip per query, membership folded into the statement the caller was already running, and nothing for callers' tests to mock. The "no groups means no group arm" special case disappears too — an empty subquery simply matches nothing. Also: _fake_note in the process tool tests had no real user_id, so its auto-MagicMock attribute reached session.get(User, ...) through the new provenance check and SQLAlchemy rejected it. The fixture now takes a real user_id defaulting to the bound caller, which makes "is this shared?" meaningful, and gains a case asserting another user's process comes back flagged. Test assertions on compiled SQL are deliberately loose about formatting: the local env has no SQLAlchemy (rule #10), so they check that the arms exist rather than guessing at exact rendering. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt	2026-07-25 22:48:01 -04:00
bvandeusenandClaude Opus 5	04b58ce01e	feat(acl): shared records are search-only and always labelled as someone else's ... Narrows the b7d6fc7 widening per the operator's call, and fixes a regression it introduced. Decision recorded as note 2094. Two scopes now, deliberately different: readable_notes_clause — everything the ACL permits, including records reached only via a direct/group note share. For EXPLICIT acts: a search the caller typed, a fetch by id. browsable_notes_clause — the caller's own records plus anything in a project they can reach. For PASSIVE surfaces: browse lists, facet counts, the process->skill manifest. The split is a trust boundary. Anything appearing unasked — in your own list, your own counts, or as a skill installed on your machine — reads as material you endorsed. A one-off someone shared with you hasn't earned that standing, so it waits until you go looking. This also dissolves the shared-Process problem by construction rather than by special case: the manifest is a passive surface, so a directly-shared Process is never installed as an auto-surfacing skill. Regression fix (#2093): b7d6fc7 widened the list queries but left the fetch path owner-only, so on the MCP path a record could be listed and then not opened — get_snippet raised not-found, get_process couldn't resolve, and the manifest emitted stubs whose get_process call would fail. snippets.get_snippet and notes.resolve_process now resolve the read scope. delete_snippet gained an explicit can_write_note guard, since being able to SEE a shared snippet must not imply being able to bin it. Provenance, so nothing arrives looking like the operator's own work: - access.describe_provenance / label_shared_items add shared/owner/permission; labelling costs no query when everything is the caller's own. - MCP: get_snippet, list_snippets, get_process and list_processes carry it, and get_process now says outright NOT to follow a shared process verbatim — its follow-as-written contract was the sharpest instance of the problem. - The skill stub for a shared Process names its author and asks for a go-ahead, instead of describing it as "the operator's saved Scribe process". - Policy stated once in the MCP _INSTRUCTIONS and the reusing-code skill: a shared record is that person's suggestion, weigh it, attribute it, ask before adopting it. - UI: shared snippets show "by <owner>" in the list and a notice above the code in the detail view, reusing SharedWithMeView's vocabulary. Plugin 0.1.15 -> 0.1.16 (skill text changed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt	2026-07-25 22:43:00 -04:00
bvandeusenandClaude Opus 4.8	b49efdcb11	refactor(scribe): retire calendar/events + person/place/list entities (backend) ... Narrow Scribe to a Claude-Code work system-of-record (milestone #194, decision note #1759). Wholesale removal per rule #22 — backend + schema half. Calendar/events + CalDAV: delete models/event, services/{events,caldav, caldav_sync}, routes/events, mcp/tools/events; strip event branches from backup (bump v3->v4), dashboard (upcoming_events), trash, recent, and the mcp server read-only allowlist + instructions. Typed entities (person/place/list): delete mcp/tools/entities; drop the notes.metadata (entity_meta) column from model/service/routes and the knowledge browse service. note_type STAYS — it also marks 'process' notes. Scheduler: event_scheduler -> recurrence_scheduler, keeping only the recurring-task spawn job (drops event reminders + CalDAV sync). Schema: migration 0069 drops the events table + notes.metadata column + orphan caldav settings rows (faithful downgrade recreates them). KEEP: recurrence.py (task recurrence), notifications task reminders, graph view, and every work surface. Frontend + plugin/docs true-up follow next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BPtbSzA4JLMAKgFZ8VTg7Q	2026-07-19 13:29:14 -04:00
bvandeusenandClaude Opus 4.8	85e0501705	feat(issues): S2 MCP tools — system CRUD + issue/system wiring ... Second slice of Issues + Systems (spec #825). New mcp/tools/systems.py: create_system, list_systems, get_system (records split into issues/tasks/notes), update_system (incl. archive via status), list_system_records (kind/open_only filters), delete_system. Registered in register_all; read tools (get_system, list_systems, list_system_records) added to the read-only-key allowlist (write tools default-deny). create_task/update_task: kind now accepts 'issue'; new system_ids (set-semantics associations) and arose_from_id (provenance, 0=unchanged/-1=clear) args. create_note/update_note: new system_ids arg (notes associate with systems too). services/notes.create_note: arose_from_id passthrough (update_note already handles it via setattr). Tests: MCP system tools + create_task issue-wiring (kind/provenance/systems), service layer mocked. Refs plan 825 (S2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-13 23:07:37 -04:00
bvandeusenandClaude Opus 4.8	b255a0f90e	refactor: rename package fabledassistant -> scribe (code-only) ... Renames src/fabledassistant -> src/scribe and all imports, plus the default DB name and DB user/password (fabled -> scribe) in config + compose. 952 refs / 154 files. Reverses the old 'internal name stays fabledassistant' convention. Code-only: live databases are still physically named 'fabledassistant'. Deployed environments must set POSTGRES_DB / POSTGRES_USER (or rename the DB) since the defaults now resolve to 'scribe'. Repo (FabledScribe), git host (fabledsword), MCP (fabled-git) and the image name (fabledscribe) are intentionally unchanged. ruff check src/ clean locally; CI (typecheck + pytest) is the gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 15:48:35 -04:00