FabledScribe/docs/architecture.md

# Architecture

## Stack

| Layer | Technology | Notes |
|-------|-----------|-------|
| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API |
| Editor | Tiptap (ProseMirror) with custom slash-command extension | |
| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files |
| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver |
| LLM | Ollama (local) | Any OpenAI-compatible API also works |
| Search | SearXNG (optional, self-hosted) | Web search + image search |
| Push | Web Push / VAPID (pywebpush 2.x) | |
| Deployment | Docker Compose | Single-container app + separate DB + LLM service |

## High-Level Component Diagram

```
┌─────────────────────────────────────────────┐
│              Docker Compose                  │
│                                              │
│  ┌──────────────────────┐   ┌────────────┐  │
│  │  scribe     │   │  ollama    │  │
│  │  ┌────────────────┐  │   │            │  │
│  │  │  Quart Server  │  │   │  LLM API   │  │
│  │  │  ┌──────────┐  │  │   │            │  │
│  │  │  │ Vue SPA  │  │  │   └────────────┘  │
│  │  │  │ (static) │  │  │         ▲         │
│  │  │  └──────────┘  │  │         │         │
│  │  │  ┌──────────┐  │  │    HTTP/REST      │
│  │  │  │ /api/*   │──┼──┼─────────┘         │
│  │  │  └──────────┘  │  │                   │
│  │  │       │        │  │  ┌────────────┐   │
│  │  │       ▼        │  │  │ PostgreSQL │   │
│  │  │  ┌──────────┐  │  │  │    16      │   │
│  │  │  │ asyncpg  │──┼──┼──▶            │   │
│  │  │  └──────────┘  │  │  └────────────┘   │
│  │  └────────────────┘  │                   │
│  └──────────────────────┘                   │
└─────────────────────────────────────────────┘
```

## Project Structure

```
scribe/
├── docker-compose.yml           # Development stack
├── docker-compose.prod.yml      # Production stack (Docker Swarm)
├── Dockerfile                   # Multi-stage build (Node → Python)
├── alembic/                     # Database migrations
│   └── versions/                # Migration files (idempotent raw SQL)
├── src/scribe/
│   ├── app.py                   # Quart app factory + blueprint registration
│   ├── config.py                # Config class (reads env vars)
│   ├── auth.py                  # login_required decorator, session checks
│   ├── models/                  # SQLAlchemy models
│   ├── mcp/                      # In-app MCP server (FastMCP, mounted at /mcp)
│   │   ├── server.py            # FastMCP instance + instructions + Quart mount
│   │   └── tools/               # Tool modules (notes, tasks, projects, rulebooks, …)
│   ├── routes/                  # API blueprints (one file per resource)
│   ├── services/                # Business logic (access, llm, tools, sharing, …)
│   └── static/                  # Built Vue SPA (generated at Docker build time)
└── frontend/
    └── src/
        ├── views/               # Page-level Vue components
        ├── components/          # Reusable UI components
        ├── composables/         # Vue composables (autosave, shortcuts, …)
        ├── stores/              # Pinia stores (auth, chat, notes, notifications, …)
        └── api/                 # Typed API client (client.ts)
```

## Key Design Decisions

**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build.

**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/<path:path>` route intercepting API GETs.

**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity.

**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching.

**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture.

**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`.

**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins.

**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend.

**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses.

**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var.

**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy.

**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs.

## Data Model

### Users

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `username` | text UNIQUE NOT NULL | |
| `email` | text nullable | |
| `password_hash` | text nullable | NULL for OAuth-only accounts |
| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier |
| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' |
| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions |
| `created_at` | timestamptz | |

### Notes (unified — includes tasks)

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `title` | text | |
| `body` | text | Markdown |
| `tags` | ARRAY[text] | GIN indexed; explicitly set by client |
| `parent_id` | int FK self nullable | Sub-tasks / sub-notes |
| `user_id` | int FK users nullable | CASCADE |
| `project_id` | int FK projects nullable | |
| `milestone_id` | int FK milestones nullable | |
| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task |
| `priority` | text nullable | `none`/`low`/`medium`/`high` |
| `due_date` | date nullable | |
| `created_at`, `updated_at` | timestamptz | |

Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`.

### Settings

Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `journal_config` (JSON: locations, temp_unit, prep schedule), `user_timezone`, `voice_*`, etc.

### Conversations / Messages

`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`journal`/`mcp`), `day_date` (YYYY-MM-DD for journal conversations), `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`.
`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`.

Title auto-generated by LLM on first exchange, re-generated every 10th message.

### Projects / Milestones

`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps.
`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps.

### Sharing & Access

`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`.

`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles.

Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission.

### Journal-Related Tables

`weather_cache`: per-user, per-`location_key` cache. Columns: `user_id`, `location_key` (`home`/`work`/etc.), `location_label`, `forecast_json` (Open-Meteo response), `previous_json` (last forecast, used to detect changes), `fetched_at`. Lat/lon are *not* stored on the cache row — they live in the user's `journal_config.locations.{home|work}` setting and are used at refresh time.

### API Keys

`api_keys`: `id`, `user_id` FK CASCADE, `prefix` (first 8 chars, displayed in UI), `key_hash` (SHA-256 of full key — full key never stored), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at`.

### App Logs

`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB.

## Detailed File Reference

### Backend (`src/scribe/`)

| File | Responsibility |
|------|---------------|
| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` |
| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper; accepts session cookie or `Authorization: Bearer <key>` |
| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods |
| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` |
| `models/note.py` | Unified Note model (notes + tasks) |
| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` |
| `models/conversation.py` | `Conversation` + `Message` models |
| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB |
| `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` |
| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) |
| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) |
| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards |
| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) |
| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm |
| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts |
| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) |
| `routes/task_logs.py` | Task work log CRUD (`GET/POST/DELETE /api/tasks/:id/logs`) |
| `routes/projects.py` | Projects CRUD + summary endpoint |
| `routes/milestones.py` | Milestones CRUD under `/api/projects/:id/milestones` |
| `routes/settings.py` | Per-user settings key-value (`GET/PUT /api/settings/:key`) |
| `routes/journal.py` | Journal config CRUD; today/day/days conversation accessors; weather endpoints (cached/current/refresh/geocode); moments CRUD; trigger-prep |
| `routes/groups.py` | Group CRUD + membership management (admin) |
| `routes/shares.py` | Share project/note with user or group; revoke; list incoming shares |
| `routes/in_app_notifications.py` | In-app notification list + mark-read + count |
| `routes/push.py` | Web Push subscription subscribe/unsubscribe; VAPID public key |
| `routes/users.py` | User profile; admin user list + delete |
| `routes/images.py` | Serve cached images at `/api/images/<id>` |
| `routes/export.py` | `GET /api/export` — personal Markdown ZIP or JSON array download |
| `routes/api_keys.py` | API key CRUD (`GET/POST/DELETE /api/api-keys`) |
| `mcp/server.py` | Mounts the in-app FastMCP server at `/mcp` (streamable HTTP, Bearer auth) |
| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation |
| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search |
| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens |
| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` |
| `services/api_keys.py` | `generate_key()`, `create_api_key()`, `list_api_keys()`, `revoke_api_key()`, `lookup_key()` (SHA-256 hash lookup) |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard |
| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` |
| ~~`services/intent.py`~~ | Removed — intent routing eliminated; the main model handles all tool routing directly |
| `services/tools/` | LLM tool package — decorator-based registry (`_registry.py`), shared helpers (`_helpers.py`), and one module per domain: `notes.py` (create/update/delete/search/list/read), `tasks.py` (list/log_work), `entities.py` (save_person/save_place/lists), `projects.py`, `calendar.py`, `web.py`, `rag.py`, `profile.py`, `rss.py`, `weather.py`, `utility.py` (calculate). `execute_tool()` and `get_tools_for_user()` are the public API. |
| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) |
| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) |
| `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management |
| `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata |
| `services/note_drafts.py` | Per-user per-note draft persistence (AI Assist pending state) |
| `services/settings.py` | `get_setting()`, `set_setting()`, `get_all_settings()` — key-value per user |
| `services/tag_suggestions.py` | `/api/notes/suggest-tags` — LLM-generated tag suggestions for note body |
| `services/access.py` | Permission resolution for all shared resources |
| `services/sharing.py` | Create/revoke shares; list shares; `get_shared_with_me()` |
| `services/groups.py` | Group CRUD; membership management |
| `services/notifications.py` | Create/read/mark-read in-app notifications |
| `services/task_logs.py` | Append/list/delete work log entries on tasks |
| `services/journal_prep.py` | Deterministic data gather (tasks/events/weather/projects/recent moments/open threads) → LLM prose opener; persisted as the first assistant message of today's journal Conversation |
| `services/journal_pipeline.py` | System-prompt builder for journal conversations; calls `build_profile_context()` so the LLM sees the user's profile + learned summary |
| `services/journal_scheduler.py` | APScheduler `BackgroundScheduler`; per-user prep job; live-reschedule via `update_user_schedule()`; catch-up logic for missed runs |
| `services/journal_search.py`, `services/moments.py` | Moment recording + search across journal history |
| `services/user_profile.py` | `build_profile_context()` consolidates profile + observations + learned_summary into a system-prompt block |
| `services/research.py` | SearXNG research pipeline: sub-queries → parallel fetch → outline → section synthesis → executive summary → index note with linked section notes |
| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools |
| `routes/events.py` | `/api/events` — event CRUD routes |
| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation |
| `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed |
| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap |
| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) |
| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup |
| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) |
| `services/email.py` | SMTP email sending for password reset; reads `SMTP_*` env vars |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache |
| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items |
| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection |

### Frontend (`frontend/src/`)

| File | Responsibility |
|------|---------------|
| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount |
| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect |
| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth |
| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` |
| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` |
| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` |
| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` |
| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` |
| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` |
| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response |
| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete |
| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver |
| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView |
| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage |
| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel |
| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management |
| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage |
| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave |
| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks |
| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" |
| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" |
| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` |
| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration |
| `components/ShareDialog.vue` | `<Teleport>` modal; user tab + group tab; current shares list |
| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over |
| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave |
| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft |
| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards |
| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm |
| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state |
| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard |
| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` |
| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` |
| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete |

## Key Services

| Service | Responsibility |
|---------|---------------|
| `services/access.py` | Permission resolution for all shared resources |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation |
| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management |
| `services/tools/` | LLM tool implementations (38 tools across 11 modules); decorator-based registry |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` |
| `services/journal_prep.py` | Data gather → LLM prose opener; persisted into today's journal conversation |
| `services/journal_scheduler.py` | APScheduler integration, per-user prep job, catch-up for missed runs |
| `services/backup.py` | Full and per-user backup export/restore (version 2 format) |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache |
| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 |

## Authentication

**Session cookies** — `HttpOnly`, `SameSite=Lax`, optionally `Secure` (`SECURE_COOKIES` env var). Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear.

**Bearer token / API keys** — `Authorization: Bearer <key>` accepted by `_check_auth()` as an alternative to session cookies. The raw key is SHA-256 hashed and looked up via `services/api_keys.py`. Keys with `scope=read` are rejected on non-safe methods (`POST`/`PATCH`/`DELETE`). Used by Fable MCP and any external API consumers.

**Local auth + OIDC/OAuth (PKCE)** — `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user.

See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions.

## LLM Pipeline Internals

### Tool Routing

No separate intent router — the main model handles all tool routing directly via Ollama's structured tool-calling output. The model receives the full tool schema list and decides whether to call a tool or respond conversationally. Extended reasoning (`think=True`) is always on for qwen3-class models: content-based gating was tried but exposed tool-call template fragility on short tool-intent prompts.

### Tool Loop

Multi-round tool loop (max 5 rounds). All implementations in `services/tools/` (decorator-based registry); `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation.

**Unified `create_note` tool** — creates both notes and tasks. Setting `status` (e.g. `"todo"`) creates a task; omitting it creates a knowledge note. All task fields (due_date, priority, milestone, parent_task, recurrence_rule) are available on the single tool.

**Duplicate protection on `create_note`:**
1. Exact title match (case-insensitive) → hard block, redirect to `update_note`
2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block
3. Semantic content similarity (threshold 0.90, body ≥ 80 chars) → soft block with `requires_confirmation: true`

**Project resolution** (`_helpers.resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55.

### Context Window and Summarisation

`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure.

History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens.

### Web Research Pipeline

`services/research.py` implements a full autonomous research pipeline:
1. Intent model generates 5 focused sub-queries
2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter)
3. Up to 15 unique URLs fetched in parallel
4. LLM generates an outline (2–8 sections with title + focus)
5. Each section synthesised in parallel from relevant sources
6. Executive summary generated from all section content
7. Index note created with executive summary + links to section notes; section notes linked back via `parent_id`
8. Falls back to single-note synthesis if outline generation fails

SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests.

### Image Cache

`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/<id>`. The user's browser never contacts the original image host.

Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB).

## RAG Pipeline

1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags.
2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each).
3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include).
4. Explicitly included notes delivered full-body.
5. `excluded_note_ids` prevents the current note from being injected as its own context.

### RAG Scope (three-value system)

`conversations.rag_project_id` controls which notes are eligible for retrieval:

| Value | Behaviour |
|-------|-----------|
| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` |
| `-1` | All notes — opt-in to global search |
| Positive int | That project's notes only |

`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths.

**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation:
- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects.
- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively.

**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup.

Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task).