FabledScribe/docs/architecture.md

# Architecture

## Stack

| Layer | Technology | Notes |
|-------|-----------|-------|
| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API |
| Editor | Tiptap (ProseMirror) with custom slash-command extension | |
| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files |
| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver |
| LLM | Ollama (local) | Any OpenAI-compatible API also works |
| Search | SearXNG (optional, self-hosted) | Web search + image search |
| Push | Web Push / VAPID (pywebpush 2.x) | |
| Deployment | Docker Compose | Single-container app + separate DB + LLM service |

## High-Level Component Diagram

```
┌─────────────────────────────────────────────┐
│              Docker Compose                  │
│                                              │
│  ┌──────────────────────┐   ┌────────────┐  │
│  │  fabledassistant     │   │  ollama    │  │
│  │  ┌────────────────┐  │   │            │  │
│  │  │  Quart Server  │  │   │  LLM API   │  │
│  │  │  ┌──────────┐  │  │   │            │  │
│  │  │  │ Vue SPA  │  │  │   └────────────┘  │
│  │  │  │ (static) │  │  │         ▲         │
│  │  │  └──────────┘  │  │         │         │
│  │  │  ┌──────────┐  │  │    HTTP/REST      │
│  │  │  │ /api/*   │──┼──┼─────────┘         │
│  │  │  └──────────┘  │  │                   │
│  │  │       │        │  │  ┌────────────┐   │
│  │  │       ▼        │  │  │ PostgreSQL │   │
│  │  │  ┌──────────┐  │  │  │    16      │   │
│  │  │  │ asyncpg  │──┼──┼──▶            │   │
│  │  │  └──────────┘  │  │  └────────────┘   │
│  │  └────────────────┘  │                   │
│  └──────────────────────┘                   │
└─────────────────────────────────────────────┘
```

## Project Structure

```
fabledassistant/
├── docker-compose.yml           # Development stack
├── docker-compose.prod.yml      # Production stack (Docker Swarm)
├── Dockerfile                   # Multi-stage build (Node → Python)
├── alembic/                     # Database migrations
│   └── versions/                # Migration files (idempotent raw SQL)
├── fable-mcp/                   # Fable MCP server package
│   └── fable_mcp/
│       ├── server.py            # FastMCP tool registrations
│       ├── client.py            # FableClient (httpx wrapper)
│       └── tools/               # Tool modules (notes, tasks, projects, …)
├── src/fabledassistant/
│   ├── app.py                   # Quart app factory + blueprint registration
│   ├── config.py                # Config class (reads env vars)
│   ├── auth.py                  # login_required decorator, session checks
│   ├── models/                  # SQLAlchemy models
│   ├── routes/                  # API blueprints (one file per resource)
│   ├── services/                # Business logic (access, llm, tools, sharing, …)
│   └── static/                  # Built Vue SPA (generated at Docker build time)
└── frontend/
    └── src/
        ├── views/               # Page-level Vue components
        ├── components/          # Reusable UI components
        ├── composables/         # Vue composables (autosave, shortcuts, …)
        ├── stores/              # Pinia stores (auth, chat, notes, notifications, …)
        └── api/                 # Typed API client (client.ts)
```

## Key Design Decisions

**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build.

**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/<path:path>` route intercepting API GETs.

**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity.

**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching.

**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture.

**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`.

**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins.

**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend.

**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses.

**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var.

**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy.

**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs.

## Data Model

### Users

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `username` | text UNIQUE NOT NULL | |
| `email` | text nullable | |
| `password_hash` | text nullable | NULL for OAuth-only accounts |
| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier |
| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' |
| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions |
| `created_at` | timestamptz | |

### Notes (unified — includes tasks)

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `title` | text | |
| `body` | text | Markdown |
| `tags` | ARRAY[text] | GIN indexed; explicitly set by client |
| `parent_id` | int FK self nullable | Sub-tasks / sub-notes |
| `user_id` | int FK users nullable | CASCADE |
| `project_id` | int FK projects nullable | |
| `milestone_id` | int FK milestones nullable | |
| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task |
| `priority` | text nullable | `none`/`low`/`medium`/`high` |
| `due_date` | date nullable | |
| `created_at`, `updated_at` | timestamptz | |

Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`.

### Settings

Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `briefing_enabled`, `briefing_locations`, `office_days`, etc.

### Conversations / Messages

`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`briefing`/`mcp`), `briefing_date`, `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`.
`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`.

Title auto-generated by LLM on first exchange, re-generated every 10th message.

### Projects / Milestones

`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps.
`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps.

### Sharing & Access

`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`.

`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles.

Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission.

### Briefing-Related Tables

`rss_feeds`: `id`, `user_id`, `url`, `name`, `category`, `last_fetched_at`.
`rss_items`: `id`, `feed_id` FK, `guid`, `title`, `url`, `summary`, `pub_date`.
`weather_cache`: per-user cache with `lat`, `lon`, `location_name`, `forecast_json`, `fetched_at`.

### API Keys

`api_keys`: `id`, `user_id` FK CASCADE, `prefix` (first 8 chars, displayed in UI), `key_hash` (SHA-256 of full key — full key never stored), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at`.

### App Logs

`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB.

## Detailed File Reference

### Backend (`src/fabledassistant/`)

| File | Responsibility |
|------|---------------|
| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` |
| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper; accepts session cookie or `Authorization: Bearer <key>` |
| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods |
| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` |
| `models/note.py` | Unified Note model (notes + tasks) |
| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` |
| `models/conversation.py` | `Conversation` + `Message` models |
| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB |
| `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` |
| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) |
| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) |
| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards |
| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) |
| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm; briefing conversation routes |
| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts |
| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) |
| `routes/task_logs.py` | Task work log CRUD (`GET/POST/DELETE /api/tasks/:id/logs`) |
| `routes/projects.py` | Projects CRUD + summary endpoint |
| `routes/milestones.py` | Milestones CRUD under `/api/projects/:id/milestones` |
| `routes/settings.py` | Per-user settings key-value (`GET/PUT /api/settings/:key`) |
| `routes/briefing.py` | Briefing conversation + reply + history; RSS feed management |
| `routes/groups.py` | Group CRUD + membership management (admin) |
| `routes/shares.py` | Share project/note with user or group; revoke; list incoming shares |
| `routes/in_app_notifications.py` | In-app notification list + mark-read + count |
| `routes/push.py` | Web Push subscription subscribe/unsubscribe; VAPID public key |
| `routes/users.py` | User profile; admin user list + delete |
| `routes/images.py` | Serve cached images at `/api/images/<id>` |
| `routes/export.py` | `GET /api/export` — personal Markdown ZIP or JSON array download |
| `routes/api_keys.py` | API key CRUD (`GET/POST/DELETE /api/api-keys`) |
| `routes/fable_mcp_dist.py` | `GET /api/fable-mcp/info` + `GET /api/fable-mcp/download` — package distribution |
| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation |
| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search |
| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens |
| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` |
| `services/api_keys.py` | `generate_key()`, `create_api_key()`, `list_api_keys()`, `revoke_api_key()`, `lookup_key()` (SHA-256 hash lookup) |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard |
| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` |
| `services/intent.py` | `classify_intent()` — fast non-streaming LLM call; intent skip heuristic; `_PRIOR_WORK_REFS` fast-path |
| `services/tools.py` | All LLM tool definitions + `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` dispatcher; duplicate guards; `_resolve_project()` 4-step lookup; `search_projects` and `set_rag_scope` tools |
| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) |
| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) |
| `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management |
| `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata |
| `services/note_drafts.py` | Per-user per-note draft persistence (AI Assist pending state) |
| `services/settings.py` | `get_setting()`, `set_setting()`, `get_all_settings()` — key-value per user |
| `services/tag_suggestions.py` | `/api/notes/suggest-tags` — LLM-generated tag suggestions for note body |
| `services/access.py` | Permission resolution for all shared resources |
| `services/sharing.py` | Create/revoke shares; list shares; `get_shared_with_me()` |
| `services/groups.py` | Group CRUD; membership management |
| `services/notifications.py` | Create/read/mark-read in-app notifications |
| `services/task_logs.py` | Append/list/delete work log entries on tasks |
| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → `GenerationBuffer` stream |
| `services/briefing_scheduler.py` | APScheduler `BackgroundScheduler`; slots with catch-up logic; async-safe via `asyncio.create_task` |
| `services/briefing_conversations.py` | Briefing conversation persistence and history queries |
| `services/briefing_profile.py` | Per-user profile note that the assistant updates over time |
| `services/research.py` | SearXNG research pipeline: 5 sub-queries → parallel fetch → synthesis; `search_images` for image category |
| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools |
| `routes/events.py` | `/api/events` — event CRUD routes |
| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation |
| `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed |
| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap |
| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) |
| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup |
| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) |
| `services/email.py` | SMTP email sending for password reset; reads `SMTP_*` env vars |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache |
| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items |
| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection |

### Frontend (`frontend/src/`)

| File | Responsibility |
|------|---------------|
| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount |
| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect |
| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth |
| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` |
| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` |
| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` |
| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` |
| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` |
| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` |
| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response |
| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete |
| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver |
| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView |
| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage |
| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel |
| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management |
| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage |
| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave |
| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks |
| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" |
| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" |
| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` |
| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration |
| `components/ShareDialog.vue` | `<Teleport>` modal; user tab + group tab; current shares list |
| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over |
| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave |
| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft |
| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards |
| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm |
| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state |
| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard |
| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` |
| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` |
| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete |

## Key Services

| Service | Responsibility |
|---------|---------------|
| `services/access.py` | Permission resolution for all shared resources |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation |
| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management |
| `services/tools.py` | All LLM tool implementations (`create_note`, `search_notes`, `get_weather`, …) |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` |
| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → briefing output |
| `services/briefing_scheduler.py` | APScheduler integration, catch-up logic for missed slots |
| `services/backup.py` | Full and per-user backup export/restore (version 2 format) |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache |
| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 |

## Authentication

**Session cookies** — `HttpOnly`, `SameSite=Lax`, optionally `Secure` (`SECURE_COOKIES` env var). Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear.

**Bearer token / API keys** — `Authorization: Bearer <key>` accepted by `_check_auth()` as an alternative to session cookies. The raw key is SHA-256 hashed and looked up via `services/api_keys.py`. Keys with `scope=read` are rejected on non-safe methods (`POST`/`PATCH`/`DELETE`). Used by Fable MCP and any external API consumers.

**Local auth + OIDC/OAuth (PKCE)** — `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user.

See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions.

## LLM Pipeline Internals

### Intent Routing

Before the main model runs, a lightweight intent classifier (`services/intent.py`) runs concurrently with `build_context()`. It makes a fast non-streaming call using a smaller dedicated model (`OLLAMA_INTENT_MODEL`, default `qwen2.5:7b`) to determine if the message requires a tool call.

**Skip heuristic** — Intent classification is skipped entirely for short messages (≤10 words) with no action/object keywords, saving 400–800ms on conversational replies.

**Prior-work fast-path** — `_PRIOR_WORK_REFS` regex detects phrases like "research you did", "note you made", "using your research" and returns no-tool immediately, preventing `search_web` from firing when the user references existing notes.

If a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk (TTFT), the tool executes, then the main model generates a follow-up with the tool result. For chat-only responses the main model streams directly.

### Tool Loop

Multi-round tool loop (max 5 rounds). All implementations in `services/tools.py`; `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation.

**Duplicate protection on `create_note` / `create_task`:**
1. Exact title match (case-insensitive) → hard block, redirect to `update_note`
2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block
3. Semantic content similarity (threshold 0.90, body ≥ 200 chars) → soft block with `requires_confirmation: true`

**Project resolution** (`_resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55.

### Context Window and Summarisation

`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure.

History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens.

### Web Research Pipeline

`services/research.py` implements a full autonomous research pipeline:
1. Intent model generates 5 focused sub-queries
2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter)
3. Up to 15 unique URLs fetched in parallel
4. Up to 12 sources passed to synthesis LLM
5. Result saved as a note with `tags=["research"]`

SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests.

### Image Cache

`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/<id>`. The user's browser never contacts the original image host.

Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB).

## RAG Pipeline

1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags.
2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each).
3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include).
4. Explicitly included notes delivered full-body.
5. `excluded_note_ids` prevents the current note from being injected as its own context.

### RAG Scope (three-value system)

`conversations.rag_project_id` controls which notes are eligible for retrieval:

| Value | Behaviour |
|-------|-----------|
| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` |
| `-1` | All notes — opt-in to global search |
| Positive int | That project's notes only |

`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths.

**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation:
- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects.
- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively.

**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup.

Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task).