docs: restructure documentation into docs/ directory, slim README, add project .mcp.json
- README.md: reduced to overview + quick start + links to docs/ - docs/architecture.md: stack, design decisions, data models, key services - docs/configuration.md: all env vars, docker-compose setup, production + security - docs/development.md: dev workflow, CI/CD, migrations, release process - docs/features.md: detailed feature breakdown + keyboard shortcuts - docs/api-keys-and-mcp.md: API key management + Fable MCP install guide - docs/sso-oauth.md: OAuth/OIDC setup (replaces docs/oauth-setup.md) - docs/changelog.md: development history from summary.md - Remove summary.md (content distributed across docs/) - Remove docs/oauth-setup.md (superseded by docs/sso-oauth.md) - .gitignore: add .mcp.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,200 @@
|
||||
# Architecture
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Technology | Notes |
|
||||
|-------|-----------|-------|
|
||||
| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API |
|
||||
| Editor | Tiptap (ProseMirror) with custom slash-command extension | |
|
||||
| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files |
|
||||
| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver |
|
||||
| LLM | Ollama (local) | Any OpenAI-compatible API also works |
|
||||
| Search | SearXNG (optional, self-hosted) | Web search + image search |
|
||||
| Push | Web Push / VAPID (pywebpush 2.x) | |
|
||||
| Deployment | Docker Compose | Single-container app + separate DB + LLM service |
|
||||
|
||||
## High-Level Component Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Docker Compose │
|
||||
│ │
|
||||
│ ┌──────────────────────┐ ┌────────────┐ │
|
||||
│ │ fabledassistant │ │ ollama │ │
|
||||
│ │ ┌────────────────┐ │ │ │ │
|
||||
│ │ │ Quart Server │ │ │ LLM API │ │
|
||||
│ │ │ ┌──────────┐ │ │ │ │ │
|
||||
│ │ │ │ Vue SPA │ │ │ └────────────┘ │
|
||||
│ │ │ │ (static) │ │ │ ▲ │
|
||||
│ │ │ └──────────┘ │ │ │ │
|
||||
│ │ │ ┌──────────┐ │ │ HTTP/REST │
|
||||
│ │ │ │ /api/* │──┼──┼─────────┘ │
|
||||
│ │ │ └──────────┘ │ │ │
|
||||
│ │ │ │ │ │ ┌────────────┐ │
|
||||
│ │ │ ▼ │ │ │ PostgreSQL │ │
|
||||
│ │ │ ┌──────────┐ │ │ │ 16 │ │
|
||||
│ │ │ │ asyncpg │──┼──┼──▶ │ │
|
||||
│ │ │ └──────────┘ │ │ └────────────┘ │
|
||||
│ │ └────────────────┘ │ │
|
||||
│ └──────────────────────┘ │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
fabledassistant/
|
||||
├── docker-compose.yml # Development stack
|
||||
├── docker-compose.prod.yml # Production stack (Docker Swarm)
|
||||
├── Dockerfile # Multi-stage build (Node → Python)
|
||||
├── alembic/ # Database migrations
|
||||
│ └── versions/ # Migration files (idempotent raw SQL)
|
||||
├── fable-mcp/ # Fable MCP server package
|
||||
│ └── fable_mcp/
|
||||
│ ├── server.py # FastMCP tool registrations
|
||||
│ ├── client.py # FableClient (httpx wrapper)
|
||||
│ └── tools/ # Tool modules (notes, tasks, projects, …)
|
||||
├── src/fabledassistant/
|
||||
│ ├── app.py # Quart app factory + blueprint registration
|
||||
│ ├── config.py # Config class (reads env vars)
|
||||
│ ├── auth.py # login_required decorator, session checks
|
||||
│ ├── models/ # SQLAlchemy models
|
||||
│ ├── routes/ # API blueprints (one file per resource)
|
||||
│ ├── services/ # Business logic (access, llm, tools, sharing, …)
|
||||
│ └── static/ # Built Vue SPA (generated at Docker build time)
|
||||
└── frontend/
|
||||
└── src/
|
||||
├── views/ # Page-level Vue components
|
||||
├── components/ # Reusable UI components
|
||||
├── composables/ # Vue composables (autosave, shortcuts, …)
|
||||
├── stores/ # Pinia stores (auth, chat, notes, notifications, …)
|
||||
└── api/ # Typed API client (client.ts)
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build.
|
||||
|
||||
**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/<path:path>` route intercepting API GETs.
|
||||
|
||||
**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity.
|
||||
|
||||
**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching.
|
||||
|
||||
**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture.
|
||||
|
||||
**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`.
|
||||
|
||||
**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins.
|
||||
|
||||
**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend.
|
||||
|
||||
**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses.
|
||||
|
||||
**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var.
|
||||
|
||||
**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy.
|
||||
|
||||
**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs.
|
||||
|
||||
## Data Model
|
||||
|
||||
### Users
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | int PK | |
|
||||
| `username` | text UNIQUE NOT NULL | |
|
||||
| `email` | text nullable | |
|
||||
| `password_hash` | text nullable | NULL for OAuth-only accounts |
|
||||
| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier |
|
||||
| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' |
|
||||
| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions |
|
||||
| `created_at` | timestamptz | |
|
||||
|
||||
### Notes (unified — includes tasks)
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | int PK | |
|
||||
| `title` | text | |
|
||||
| `body` | text | Markdown |
|
||||
| `tags` | ARRAY[text] | GIN indexed; explicitly set by client |
|
||||
| `parent_id` | int FK self nullable | Sub-tasks / sub-notes |
|
||||
| `user_id` | int FK users nullable | CASCADE |
|
||||
| `project_id` | int FK projects nullable | |
|
||||
| `milestone_id` | int FK milestones nullable | |
|
||||
| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task |
|
||||
| `priority` | text nullable | `none`/`low`/`medium`/`high` |
|
||||
| `due_date` | date nullable | |
|
||||
| `created_at`, `updated_at` | timestamptz | |
|
||||
|
||||
Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`.
|
||||
|
||||
### Settings
|
||||
|
||||
Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `briefing_enabled`, `briefing_locations`, `office_days`, etc.
|
||||
|
||||
### Conversations / Messages
|
||||
|
||||
`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`briefing`/`mcp`), `briefing_date`, `created_at`, `updated_at`.
|
||||
`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`.
|
||||
|
||||
Title auto-generated by LLM on first exchange, re-generated every 10th message.
|
||||
|
||||
### Projects / Milestones
|
||||
|
||||
`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, timestamps.
|
||||
`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps.
|
||||
|
||||
### Sharing & Access
|
||||
|
||||
`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`.
|
||||
|
||||
`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles.
|
||||
|
||||
Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission.
|
||||
|
||||
### Briefing-Related Tables
|
||||
|
||||
`rss_feeds`: `id`, `user_id`, `url`, `name`, `category`, `last_fetched_at`.
|
||||
`rss_items`: `id`, `feed_id` FK, `guid`, `title`, `url`, `summary`, `pub_date`.
|
||||
`weather_cache`: per-user cache with `lat`, `lon`, `location_name`, `forecast_json`, `fetched_at`.
|
||||
|
||||
### App Logs
|
||||
|
||||
`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB.
|
||||
|
||||
## Key Services
|
||||
|
||||
| Service | Responsibility |
|
||||
|---------|---------------|
|
||||
| `services/access.py` | Permission resolution for all shared resources |
|
||||
| `services/llm.py` | `build_context()`, RAG injection, history summarisation |
|
||||
| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management |
|
||||
| `services/tools.py` | All LLM tool implementations (`create_note`, `search_notes`, `get_weather`, …) |
|
||||
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` |
|
||||
| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → briefing output |
|
||||
| `services/briefing_scheduler.py` | APScheduler integration, catch-up logic for missed slots |
|
||||
| `services/backup.py` | Full and per-user backup export/restore (version 2 format) |
|
||||
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache |
|
||||
| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 |
|
||||
|
||||
## Authentication
|
||||
|
||||
Local username/password auth and OIDC/OAuth (PKCE). `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user.
|
||||
|
||||
Session cookies: `HttpOnly`, `SameSite=Lax`, optionally `Secure`. Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear.
|
||||
|
||||
See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions.
|
||||
|
||||
## RAG Pipeline
|
||||
|
||||
1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call.
|
||||
2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each).
|
||||
3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include).
|
||||
4. Explicitly included notes delivered full-body.
|
||||
5. Project-scoped RAG: `rag_project_id` param restricts all searches to a project's notes.
|
||||
6. `excluded_note_ids` prevents the current note from being injected as its own context.
|
||||
|
||||
Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task).
|
||||
Reference in New Issue
Block a user