From 87c55691fb674f5b57b6527b23301ad7292388e9 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 25 Mar 2026 18:16:41 -0400 Subject: [PATCH] docs: update architecture and features for calendar + RAG scoping - architecture.md: add Event model, services/events.py, routes/events.py, CalendarView.vue, EventSlideOver.vue; update Conversation and Project data model tables with new columns; update RAG Pipeline section with three-value scope system, search_projects/set_rag_scope tools, and project summary background job; fix dead-code note for models/event.py; update execute_tool() signature docs; add services/projects.py entry - features.md: replace CalDAV section with full Calendar section covering internal store, AI tools, HomeView widget, and optional CalDAV sync; update AI Chat scope chip description; remove done "Calendar view" item from roadmap Co-Authored-By: Claude Sonnet 4.6 --- docs/architecture.md | 44 +++++++++++++++++++++++++++++++++----------- docs/features.md | 15 +++++++++++---- 2 files changed, 44 insertions(+), 15 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index d6ee9fd..91ab316 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -137,14 +137,14 @@ Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/sett ### Conversations / Messages -`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`briefing`/`mcp`), `briefing_date`, `created_at`, `updated_at`. +`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`briefing`/`mcp`), `briefing_date`, `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`. `messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`. Title auto-generated by LLM on first exchange, re-generated every 10th message. ### Projects / Milestones -`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, timestamps. +`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps. `milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps. ### Sharing & Access @@ -184,7 +184,7 @@ Permission resolution is centralised in `services/access.py`. `get_project_permi | `models/conversation.py` | `Conversation` + `Message` models | | `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB | | `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` | -| `models/event.py` | Dead code — CalDAV/Radicale internal events table. Radicale was trialled and removed; `calendar_sync.py` is also dead. | +| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) | | `routes/api.py` | `/api` blueprint; `GET /api/health` (public) | | `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards | | `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) | @@ -213,8 +213,9 @@ Permission resolution is centralised in `services/access.py`. `get_project_permi | `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard | | `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` | | `services/intent.py` | `classify_intent()` — fast non-streaming LLM call; intent skip heuristic; `_PRIOR_WORK_REFS` fast-path | -| `services/tools.py` | All LLM tool definitions + `execute_tool()` dispatcher; duplicate guards; `_resolve_project()` 4-step lookup | -| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` (pgvector cosine similarity) | +| `services/tools.py` | All LLM tool definitions + `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` dispatcher; duplicate guards; `_resolve_project()` 4-step lookup; `search_projects` and `set_rag_scope` tools | +| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) | +| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) | | `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) | | `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management | | `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata | @@ -231,7 +232,9 @@ Permission resolution is centralised in `services/access.py`. `get_project_permi | `services/briefing_conversations.py` | Briefing conversation persistence and history queries | | `services/briefing_profile.py` | Per-user profile note that the assistant updates over time | | `services/research.py` | SearXNG research pipeline: 5 sub-queries → parallel fetch → synthesis; `search_images` for image category | -| `services/caldav.py` | Full CalDAV event/todo lifecycle; synchronous library calls in asyncio executor | +| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools | +| `routes/events.py` | `/api/events` — event CRUD routes | +| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation | | `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed | | `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap | | `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) | @@ -256,7 +259,9 @@ Permission resolution is centralised in `services/access.py`. `get_project_permi | `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` | | `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` | | `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response | -| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; model selector; message queue; bulk delete | +| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete | +| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver | +| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView | | `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage | | `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel | | `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management | @@ -318,7 +323,7 @@ If a tool is detected, the intent's one-sentence `ack` field is streamed as the ### Tool Loop -Multi-round tool loop (max 5 rounds). All implementations in `services/tools.py`; `execute_tool()` is the dispatcher. +Multi-round tool loop (max 5 rounds). All implementations in `services/tools.py`; `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation. **Duplicate protection on `create_note` / `create_task`:** 1. Exact title match (case-insensitive) → hard block, redirect to `update_note` @@ -352,11 +357,28 @@ Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 ## RAG Pipeline -1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call. +1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags. 2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each). 3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include). 4. Explicitly included notes delivered full-body. -5. Project-scoped RAG: `rag_project_id` param restricts all searches to a project's notes. -6. `excluded_note_ids` prevents the current note from being injected as its own context. +5. `excluded_note_ids` prevents the current note from being injected as its own context. + +### RAG Scope (three-value system) + +`conversations.rag_project_id` controls which notes are eligible for retrieval: + +| Value | Behaviour | +|-------|-----------| +| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` | +| `-1` | All notes — opt-in to global search | +| Positive int | That project's notes only | + +`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths. + +**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation: +- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects. +- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively. + +**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup. Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task). diff --git a/docs/features.md b/docs/features.md index a9cbe7f..6c1ad9a 100644 --- a/docs/features.md +++ b/docs/features.md @@ -43,7 +43,7 @@ Tasks carry status (`todo` → `in_progress` → `done`), priority (`none`/`low` Full conversation history with SSE streaming. Features: - **RAG** — Semantically relevant notes (≥ 0.60 cosine similarity) auto-injected as context. Notes 0.45–0.60 shown in sidebar as "Suggested." - **Attach notes** — Paperclip icon to include specific notes in context. -- **Project scope** — Restrict RAG to a single project's notes via the sidebar selector. +- **RAG scope chip** — Pill above the input bar shows the current note scope. Click to switch: "Orphan notes only" (default — project notes stay out of general chat), any active project, or "All notes." Scope is persisted per conversation. The AI can also call `search_projects` and `set_rag_scope` mid-conversation to switch scope automatically; the chip pulses when this happens. - **Tool calls** — The assistant can create/update notes, tasks, projects, milestones, search the web, check weather, read RSS, query calendar events, and more. Tool calls display inline with confirm/deny for creates. - **Thinking mode** — Toggle extended reasoning for complex questions. - **Abort** — Stop button cancels in-flight generation. @@ -70,9 +70,17 @@ Full conversation history with SSE streaming. Features: The assistant can search the web (SearXNG) and fetch pages, synthesising findings into notes. A lightweight `search_web` tool answers quick questions inline without saving. Requires `SEARXNG_URL` to be configured. -## CalDAV Calendar +## Calendar -Connect an external CalDAV server (Nextcloud, Radicale, etc.) and manage calendar events via natural language. Tools: `create_event`, `list_events`, `search_events`, `update_event`, `delete_event`. Activated when user configures CalDAV credentials in Settings → Integrations. +`/calendar` shows a full FullCalendar view (month, week, day). Click an empty slot to create an event; click an existing event to edit or delete it via a slide-over panel. + +**Internal events store** — Events are stored in the app database (`events` table), making them available without any external calendar. Fields: title, description, start/end datetime, all-day toggle, location, colour. + +**AI tools** — `create_event`, `list_events`, `search_events`, `update_event`, `delete_event` all operate on the internal store. Tool-call result cards in chat are clickable and open the same EventSlideOver for editing. + +**HomeView widget** — The dashboard shows today's and the next 7 days' events as clickable cards above the hero project. + +**CalDAV sync (optional)** — Connect an external CalDAV server (Nextcloud, Radicale, etc.) in Settings → Integrations. Events sync bidirectionally via a `caldav_uid` field. ## Sharing and Collaboration @@ -119,7 +127,6 @@ Settings are tabbed: ## Roadmap -- Calendar/timeline view for tasks with due dates - Email integration (read/send via IMAP/SMTP tools in chat) - Session invalidation on user deletion - Flutter push notifications (requires FCM/APNs — separate from web VAPID)