bvandeusen/FabledGrimoire
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases 53 Wiki Activity

docs: architecture, configuration, development and features describe Scribe as it is today

Browse Source 
Milestone 544, step 6217 (follow-up to #1778, which rewrote features.md and
api-reference.md in July but left architecture.md and configuration.md alone).

- architecture.md, configuration.md, development.md: rewritten against the
  code. They still described Ollama, chat, SSE generation, voice, project
  summaries, image caching, Docker secrets that the compose files don't use,
  a raw-SQL migration convention the migrations no longer follow, a 27-entry
  migration list, CalVer YY.MM.DD.N tags and the removed infra/ files.
- features.md: adds what shipped since July (spikes, claims, moments, rule
  versions and verification, lessons, reply shapes, snippets and the shape
  ledger, forge integration, family canon, design systems, retrieval tuning)
  and corrects the Settings tabs and keyboard shortcuts (g g is the graph).
- api-reference.md: adds the endpoints for snippets, lessons, design
  systems, family canon, platforms, shared areas, retrieval, rule versions,
  relations and verification, forge connections, webhooks, scheduled
  backups and the newer plugin hooks.
- api-keys-and-mcp.md: Settings tab is "MCP Access"; a plugin install
  section; the tool-family table matches the tools that exist.
- design-system.md: the dated "decisions in progress" changelog (April
  PR log, chat/journal/calendar surfaces) becomes a short present-tense
  section; the stored design system is named as the authority.
- sso-oauth.md: the app is Scribe, not "Fabled Assistant".
- README: feature summary and doc table brought current.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-21 19:23:56 -04:00
co-authored by Claude Opus 5.5
parent 82e49176b1
commit 47d40bffd8
10 changed files with 410 additions and 964 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+67 -352
View File
@@ -1,380 +1,95 @@
# Architecture
Scribe is one app container and one PostgreSQL container. There is no language model in the stack: Claude works with Scribe from outside, through the MCP endpoint and the Claude Code plugin, and people use the web UI.
## Stack
| Layer | Technology | Notes |
|-------|-----------|-------|
| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API |
| Editor | Tiptap (ProseMirror) with custom slash-command extension | |
| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files |
| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver |
| LLM | Ollama (local) | Any OpenAI-compatible API also works |
| Search | SearXNG (optional, self-hosted) | Web search + image search |
| Push | Web Push / VAPID (pywebpush 2.x) | |
| Deployment | Docker Compose | Single-container app + separate DB + LLM service |
| Layer | Technology |
|-------|-----------|
| Frontend | Vue 3, TypeScript, Vite, Pinia, Vue Router; Tiptap for the editor |
| Backend | Python 3.14, Quart (async), served by Hypercorn |
| MCP | The `mcp` SDK's server, mounted in the same app at `/mcp` (streamable HTTP) |
| Database | PostgreSQL 17 with pgvector; SQLAlchemy 2 (async, asyncpg); Alembic migrations |
| Embeddings | fastembed running in-process (`BAAI/bge-small-en-v1.5`, 384 dimensions) |
| Plugin | Claude Code hooks (POSIX shell + awk), skills and a slash command, shipped from this repo |
## High-Level Component Diagram
## How the pieces fit
```
┌─────────────────────────────────────────────┐
│ Docker Compose │
│ │
│ ┌──────────────────────┐ ┌────────────┐ │
│ │ scribe │ │ ollama │ │
│ │ ┌────────────────┐ │ │ │ │
│ │ │ Quart Server │ │ │ LLM API │ │
│ │ │ ┌──────────┐ │ │ │ │ │
│ │ │ │ Vue SPA │ │ │ └────────────┘ │
│ │ │ │ (static) │ │ │ ▲ │
│ │ │ └──────────┘ │ │ │ │
│ │ │ ┌──────────┐ │ │ HTTP/REST │
│ │ │ │ /api/* │──┼──┼─────────┘ │
│ │ │ └──────────┘ │ │ │
│ │ │ │ │ │ ┌────────────┐ │
│ │ │ ▼ │ │ │ PostgreSQL │ │
│ │ │ ┌──────────┐ │ │ │ 16 │ │
│ │ │ │ asyncpg │──┼──┼──▶ │ │
│ │ │ └──────────┘ │ │ └────────────┘ │
│ │ └────────────────┘ │ │
│ └──────────────────────┘ │
└─────────────────────────────────────────────┘
Claude Code ──── plugin hooks ──── /api/plugin/* ┐
│ │
└────────── MCP tools ──────── /mcp ├── Quart app ── PostgreSQL + pgvector
│ │
Browser ──────── Vue SPA ────────── /api/* ┘ └── in-process jobs and embedder
```
## Project Structure
- **The web UI** is a Vue single-page app, built into the image and served by Quart. Every unknown path that isn't under `/api/` falls back to `index.html`; `/api/*` always answers JSON.
- **The REST API** (`src/scribe/routes/`) backs the UI, one blueprint per resource.
- **The MCP server** (`src/scribe/mcp/`) is how Claude reads and writes records. Each module in `mcp/tools/` registers its tools, and the server's instructions and each tool's docstring are the contract an agent reads. Requests authenticate with an API key as a Bearer token.
- **The plugin** (`plugin/`) runs in the user's Claude Code. Its hooks call `/api/plugin/*` to put the right context in front of Claude at the right moment: the project at session start, matching rules and prior art before a prompt or a tool call, the rules mounted on a moment of work, and a check on the reply that ends a turn. Its skills describe the working practices. See [API Keys & MCP](api-keys-and-mcp.md) to install it.
- **Services** (`src/scribe/services/`) hold the logic both doors share. Routes and MCP tools stay thin: parse the request, check access, call a service, return a plain dict.
```
scribe/
├── docker-compose.yml # Development stack
├── docker-compose.prod.yml # Production stack (Docker Swarm)
├── Dockerfile # Multi-stage build (Node → Python)
├── alembic/ # Database migrations
│ └── versions/ # Migration files (idempotent raw SQL)
├── src/scribe/
│ ├── app.py # Quart app factory + blueprint registration
│ ├── config.py # Config class (reads env vars)
│ ├── auth.py # login_required decorator, session checks
│ ├── models/ # SQLAlchemy models
│ ├── mcp/ # In-app MCP server (FastMCP, mounted at /mcp)
│ │ ├── server.py # FastMCP instance + instructions + Quart mount
│ │ └── tools/ # Tool modules (notes, tasks, projects, rulebooks, …)
│ ├── routes/ # API blueprints (one file per resource)
│ ├── services/ # Business logic (access, llm, tools, sharing, …)
│ └── static/ # Built Vue SPA (generated at Docker build time)
└── frontend/
└── src/
├── views/ # Page-level Vue components
├── components/ # Reusable UI components
├── composables/ # Vue composables (autosave, shortcuts, …)
├── stores/ # Pinia stores (auth, chat, notes, notifications, …)
└── api/ # Typed API client (client.ts)
```
## Data model
## Key Design Decisions
Most records are rows in one `notes` table, told apart by kind:
**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build.
- **Notes**, and the specialised kinds that are notes underneath: **snippets** (recorded prior art, with a canonical code location), **lessons** (transferable insights, linked to the rules they instance) and **processes** (stored procedures, synced to Claude Code as skills).
- **Tasks**: a note with a status. A task's kind is `work`, `issue` or `spike`. Tasks carry work logs, sub-tasks, recurrence and a session claim.
**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/<path:path>` route intercepting API GETs.
Around them:
**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity.
- **Projects** group work and carry a goal. **Milestones** are plans: a design body plus ordered step tasks.
- **Systems** are a project's named areas; records are tagged with them, and each maps to a canonical system shared across projects.
- **Rulebooks → topics → rules.** A rule in a rulebook is global; a rule on a project applies there only. Rules have a trigger (`when_to_apply`), versions, relations to other rules, and can be mounted on **moments** of work (`work.change`, `reply.report`, ...), which plugin actions reach through moment mappings.
- **Family canon**: ideas shared across projects on the same platform, and each project's answer to them (adopted, variant, owed).
- **Design systems**: tokens and guidance, with inheritance; the stylesheet is generated from them.
- **The shape ledger**: code shapes extracted from bound repos and their classification against snippets.
- **Embeddings** for notes (chunked), rules, milestones and systems, in pgvector columns.
- **Telemetry**: retrieval logs, note and rule usage, and the tuning history of retrieval thresholds.
- **Users, groups, shares and API keys.** Access is decided in the app (`services/access.py`): owners, direct shares and group shares, with read or write permission.
**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching.
Deleted records go to the trash first and can be restored until retention purges them.
**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture.
## Retrieval
**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`.
Search and the plugin's context injection run on the same pipeline (`services/retrieval_pipeline.py`): embed the query, rank stored chunks by cosine similarity in pgvector, keep what clears the surface's threshold, and log what was shown. Each surface (MCP search, auto-inject, the pre-tool rule arm, the write-path prior-art arm, ...) is a spec over those stages with its own thresholds and budget. The thresholds are settings an admin can tune, and every change is recorded with its reason. When the embedder is unavailable, search falls back to keyword matching.
**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins.
The embedding model downloads into `FASTEMBED_CACHE_DIR` (default `/data/fastembed-cache`) on first use, so the `/data` volume keeps it across restarts.
**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend.
## Background work
**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses.
All in-process; there is no separate worker container.
**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var.
- At startup the app records any shipped default that moved in this release, starts its schedulers, and only then backfills missing embeddings, so the backfill never competes with startup for the database.
- Recurring tasks spawn every 15 minutes. Daily jobs: version auto-pinning (03:00 UTC), trash purge (03:30 UTC), targeted `VACUUM ANALYZE` (04:00 UTC by default). Scheduled backups check hourly whether one is due ([Backups](backups.md)).
- Log, notification and auth-token retention run on their own loops.
- A diagnostics heartbeat writes to `/data/diagnostics`.
**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy.
**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs.
## Data Model
### Users
| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `username` | text UNIQUE NOT NULL | |
| `email` | text nullable | |
| `password_hash` | text nullable | NULL for OAuth-only accounts |
| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier |
| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' |
| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions |
| `created_at` | timestamptz | |
### Notes (unified — includes tasks)
| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `title` | text | |
| `body` | text | Markdown |
| `tags` | ARRAY[text] | GIN indexed; explicitly set by client |
| `parent_id` | int FK self nullable | Sub-tasks / sub-notes |
| `user_id` | int FK users nullable | CASCADE |
| `project_id` | int FK projects nullable | |
| `milestone_id` | int FK milestones nullable | |
| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task |
| `priority` | text nullable | `none`/`low`/`medium`/`high` |
| `due_date` | date nullable | |
| `created_at`, `updated_at` | timestamptz | |
Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`.
### Settings
Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `journal_config` (JSON: locations, temp_unit, prep schedule), `user_timezone`, `voice_*`, etc.
### Conversations / Messages
`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`journal`/`mcp`), `day_date` (YYYY-MM-DD for journal conversations), `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`.
`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`.
Title auto-generated by LLM on first exchange, re-generated every 10th message.
### Projects / Milestones
`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps.
`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps.
### Sharing & Access
`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`.
`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles.
Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission.
### Journal-Related Tables
`weather_cache`: per-user, per-`location_key` cache. Columns: `user_id`, `location_key` (`home`/`work`/etc.), `location_label`, `forecast_json` (Open-Meteo response), `previous_json` (last forecast, used to detect changes), `fetched_at`. Lat/lon are *not* stored on the cache row — they live in the user's `journal_config.locations.{home|work}` setting and are used at refresh time.
### API Keys
`api_keys`: `id`, `user_id` FK CASCADE, `prefix` (first 8 chars, displayed in UI), `key_hash` (SHA-256 of full key — full key never stored), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at`.
### App Logs
`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB.
## Detailed File Reference
### Backend (`src/scribe/`)
| File | Responsibility |
|------|---------------|
| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` |
| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper; accepts session cookie or `Authorization: Bearer <key>` |
| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods |
| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` |
| `models/note.py` | Unified Note model (notes + tasks) |
| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` |
| `models/conversation.py` | `Conversation` + `Message` models |
| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB |
| `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` |
| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) |
| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) |
| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards |
| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) |
| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm |
| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts |
| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) |
| `routes/task_logs.py` | Task work log CRUD (`GET/POST/DELETE /api/tasks/:id/logs`) |
| `routes/projects.py` | Projects CRUD + summary endpoint |
| `routes/milestones.py` | Milestones CRUD under `/api/projects/:id/milestones` |
| `routes/settings.py` | Per-user settings key-value (`GET/PUT /api/settings/:key`) |
| `routes/journal.py` | Journal config CRUD; today/day/days conversation accessors; weather endpoints (cached/current/refresh/geocode); moments CRUD; trigger-prep |
| `routes/groups.py` | Group CRUD + membership management (admin) |
| `routes/shares.py` | Share project/note with user or group; revoke; list incoming shares |
| `routes/in_app_notifications.py` | In-app notification list + mark-read + count |
| `routes/push.py` | Web Push subscription subscribe/unsubscribe; VAPID public key |
| `routes/users.py` | User profile; admin user list + delete |
| `routes/images.py` | Serve cached images at `/api/images/<id>` |
| `routes/export.py` | `GET /api/export` — personal Markdown ZIP or JSON array download |
| `routes/api_keys.py` | API key CRUD (`GET/POST/DELETE /api/api-keys`) |
| `mcp/server.py` | Mounts the in-app FastMCP server at `/mcp` (streamable HTTP, Bearer auth) |
| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation |
| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search |
| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens |
| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` |
| `services/api_keys.py` | `generate_key()`, `create_api_key()`, `list_api_keys()`, `revoke_api_key()`, `lookup_key()` (SHA-256 hash lookup) |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard |
| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` |
| ~~`services/intent.py`~~ | Removed — intent routing eliminated; the main model handles all tool routing directly |
| `services/tools/` | LLM tool package — decorator-based registry (`_registry.py`), shared helpers (`_helpers.py`), and one module per domain: `notes.py` (create/update/delete/search/list/read), `tasks.py` (list/log_work), `entities.py` (save_person/save_place/lists), `projects.py`, `calendar.py`, `web.py`, `rag.py`, `profile.py`, `rss.py`, `weather.py`, `utility.py` (calculate). `execute_tool()` and `get_tools_for_user()` are the public API. |
| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) |
| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) |
| `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management |
| `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata |
| `services/note_drafts.py` | Per-user per-note draft persistence (AI Assist pending state) |
| `services/settings.py` | `get_setting()`, `set_setting()`, `get_all_settings()` — key-value per user |
| `services/tag_suggestions.py` | `/api/notes/suggest-tags` — LLM-generated tag suggestions for note body |
| `services/access.py` | Permission resolution for all shared resources |
| `services/sharing.py` | Create/revoke shares; list shares; `get_shared_with_me()` |
| `services/groups.py` | Group CRUD; membership management |
| `services/notifications.py` | Create/read/mark-read in-app notifications |
| `services/task_logs.py` | Append/list/delete work log entries on tasks |
| `services/journal_prep.py` | Deterministic data gather (tasks/events/weather/projects/recent moments/open threads) → LLM prose opener; persisted as the first assistant message of today's journal Conversation |
| `services/journal_pipeline.py` | System-prompt builder for journal conversations; calls `build_profile_context()` so the LLM sees the user's profile + learned summary |
| `services/journal_scheduler.py` | APScheduler `BackgroundScheduler`; per-user prep job; live-reschedule via `update_user_schedule()`; catch-up logic for missed runs |
| `services/journal_search.py`, `services/moments.py` | Moment recording + search across journal history |
| `services/user_profile.py` | `build_profile_context()` consolidates profile + observations + learned_summary into a system-prompt block |
| `services/research.py` | SearXNG research pipeline: sub-queries → parallel fetch → outline → section synthesis → executive summary → index note with linked section notes |
| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools |
| `routes/events.py` | `/api/events` — event CRUD routes |
| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation |
| `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed |
| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap |
| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) |
| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup |
| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) |
| `services/email.py` | SMTP email sending for password reset; reads `SMTP_*` env vars |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache |
| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items |
| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection |
### Frontend (`frontend/src/`)
| File | Responsibility |
|------|---------------|
| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount |
| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect |
| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth |
| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` |
| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` |
| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` |
| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` |
| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` |
| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` |
| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response |
| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete |
| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver |
| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView |
| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage |
| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel |
| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management |
| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage |
| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave |
| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks |
| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" |
| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" |
| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` |
| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration |
| `components/ShareDialog.vue` | `<Teleport>` modal; user tab + group tab; current shares list |
| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over |
| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave |
| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft |
| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards |
| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm |
| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state |
| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard |
| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` |
| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` |
| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete |
## Key Services
| Service | Responsibility |
|---------|---------------|
| `services/access.py` | Permission resolution for all shared resources |
| `services/llm.py` | `build_context()`, RAG injection, history summarisation |
| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management |
| `services/tools/` | LLM tool implementations (38 tools across 11 modules); decorator-based registry |
| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` |
| `services/journal_prep.py` | Data gather → LLM prose opener; persisted into today's journal conversation |
| `services/journal_scheduler.py` | APScheduler integration, per-user prep job, catch-up for missed runs |
| `services/backup.py` | Full and per-user backup export/restore (version 2 format) |
| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache |
| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 |
Every database wait on startup has a deadline. Migrations wait for the database to accept connections (up to five minutes) before they run, so the app starts cleanly whatever order the containers come up in.
## Authentication
**Session cookies** — `HttpOnly`, `SameSite=Lax`, optionally `Secure` (`SECURE_COOKIES` env var). Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear.
- **Sessions**: `HttpOnly`, `SameSite=Lax` cookies, `Secure` when `SECURE_COOKIES=true`. Changing a password bumps the user's session version and signs out their other sessions.
- **API keys**: `Authorization: Bearer fmcp_...`, stored hashed. A `read` key may only call read endpoints and read MCP tools.
- **OIDC** single sign-on with PKCE, optionally replacing local logins ([SSO / OAuth](sso-oauth.md)).
- **Rate limits** on register, login and forgot-password, per client IP (`TRUST_PROXY_HEADERS` reads the IP from the proxy's headers).
**Bearer token / API keys** — `Authorization: Bearer <key>` accepted by `_check_auth()` as an alternative to session cookies. The raw key is SHA-256 hashed and looked up via `services/api_keys.py`. Keys with `scope=read` are rejected on non-safe methods (`POST`/`PATCH`/`DELETE`). Used by Fable MCP and any external API consumers.
Credentials Scribe stores for outside services (forge tokens, the backup destination's S3 secret) are sealed under a key derived from `SECRET_KEY`.
**Local auth + OIDC/OAuth (PKCE)** — `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user.
## Repository layout
See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions.
## LLM Pipeline Internals
### Tool Routing
No separate intent router — the main model handles all tool routing directly via Ollama's structured tool-calling output. The model receives the full tool schema list and decides whether to call a tool or respond conversationally. Extended reasoning (`think=True`) is always on for qwen3-class models: content-based gating was tried but exposed tool-call template fragility on short tool-intent prompts.
### Tool Loop
Multi-round tool loop (max 5 rounds). All implementations in `services/tools/` (decorator-based registry); `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation.
**Unified `create_note` tool** — creates both notes and tasks. Setting `status` (e.g. `"todo"`) creates a task; omitting it creates a knowledge note. All task fields (due_date, priority, milestone, parent_task, recurrence_rule) are available on the single tool.
**Duplicate protection on `create_note`:**
1. Exact title match (case-insensitive) → hard block, redirect to `update_note`
2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block
3. Semantic content similarity (threshold 0.90, body ≥ 80 chars) → soft block with `requires_confirmation: true`
**Project resolution** (`_helpers.resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55.
### Context Window and Summarisation
`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure.
History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens.
### Web Research Pipeline
`services/research.py` implements a full autonomous research pipeline:
1. Intent model generates 5 focused sub-queries
2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter)
3. Up to 15 unique URLs fetched in parallel
4. LLM generates an outline (2–8 sections with title + focus)
5. Each section synthesised in parallel from relevant sources
6. Executive summary generated from all section content
7. Index note created with executive summary + links to section notes; section notes linked back via `parent_id`
8. Falls back to single-note synthesis if outline generation fails
SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests.
### Image Cache
`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/<id>`. The user's browser never contacts the original image host.
Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB).
## RAG Pipeline
1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags.
2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each).
3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include).
4. Explicitly included notes delivered full-body.
5. `excluded_note_ids` prevents the current note from being injected as its own context.
### RAG Scope (three-value system)
`conversations.rag_project_id` controls which notes are eligible for retrieval:
| Value | Behaviour |
|-------|-----------|
| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` |
| `-1` | All notes — opt-in to global search |
| Positive int | That project's notes only |
`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths.
**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation:
- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects.
- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively.
**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup.
Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task).
```
├── alembic/versions/ database migrations
├── src/scribe/
│ ├── app.py app factory, blueprints, startup and shutdown
│ ├── config.py environment configuration
│ ├── models/ SQLAlchemy models
│ ├── routes/ REST blueprints
│ ├── mcp/ MCP server and its tool modules
│ ├── services/ shared logic
│ └── restore.py `python -m scribe.restore` for a fresh instance
├── frontend/src/ views, components, stores, composables, api client
├── plugin/ Claude Code hooks, skills and commands
├── scripts/ CI checks and the plugin version minter
└── tests/ unit tests, plus `-m integration` against real Postgres
```