diff --git a/Makefile b/Makefile index 2c74805f..cb7476ef 100644 --- a/Makefile +++ b/Makefile @@ -20,7 +20,7 @@ health: migrate: docker compose exec app alembic upgrade head -# --- Local dev (requires: pip install -e ".[dev]" and cd frontend && npm ci) --- +# --- Local dev (requires: uv sync --extra dev and cd frontend && npm ci) --- lint: ruff check src/ diff --git a/README.md b/README.md index 48c10735..1050f18c 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A self-hosted work system-of-record for software projects, built to be driven by ## Features -Notes and tasks with a Markdown editor, sub-tasks, milestones, issues, and kanban project workspaces. Stored processes, an engineering rulebook system (with an inception step that decides what each project inherits), and semantic search with proactive knowledge-injection into Claude's context. A knowledge graph, per-user/group sharing, and a built-in MCP server (`/mcp`) plus a bundled Claude Code plugin so Claude can record and recall your work directly. +Notes, tasks and issues; projects with milestones as plans; rules that reach Claude when its work matches them or at a moment of work; lessons, snippets of prior art and a ledger of the code they account for; design systems; ideas shared across projects on the same platform; and stored processes. Semantic search feeds both Claude's recall and the plugin's context injection. A knowledge graph, sharing with users and groups, and scheduled backups round it out. See [Features](docs/features.md). ## Quick Start @@ -33,6 +33,8 @@ Open `http://localhost:5000`. The first user to register becomes admin. To conne | [Development](docs/development.md) | Dev workflow, CI/CD, migrations, release process | | [API Keys & MCP](docs/api-keys-and-mcp.md) | API keys and installing the Claude Code plugin | | [SSO / OAuth](docs/sso-oauth.md) | OIDC setup for Authentik, Keycloak, and other providers | +| [Backups](docs/backups.md) | Scheduled backups to a folder or S3, and restoring from one | +| [Design system](docs/design-system.md) | The FabledSword house style and how Scribe applies it | | [API Reference](docs/api-reference.md) | All REST API endpoints | ## License diff --git a/docs/api-keys-and-mcp.md b/docs/api-keys-and-mcp.md index ad4aa476..e260d0fc 100644 --- a/docs/api-keys-and-mcp.md +++ b/docs/api-keys-and-mcp.md @@ -2,7 +2,7 @@ ## API Keys -API keys let external tools access your Fable data without a browser session. Each key is scoped to a single user — it can only access data that user owns or has been shared with them. +API keys let external tools access your Scribe data without a browser session. Each key is scoped to a single user — it can only access data that user owns or has been shared with them. ### Scopes @@ -15,7 +15,7 @@ Admin-level operations (log access, user management) require a `write`-scoped ke ### Creating a Key -1. Go to **Settings → API Keys** +1. Go to **Settings → MCP Access** 2. Enter a name (e.g. "Claude MCP", "Home Server") 3. Choose scope 4. Click **Generate Key** @@ -26,7 +26,7 @@ config (see **Scribe MCP Server** below). ### Revoking a Key -Click **Revoke** next to the key in the API Keys table and confirm. Revoked keys are deleted immediately. +Click **Revoke** next to the key in the keys table and confirm. Revoked keys are deleted immediately. --- @@ -41,14 +41,25 @@ client straight to the URL with a Bearer token. ### Authentication -Authenticate with an API key generated from **Settings → API Keys** (see above), +Authenticate with an API key generated from **Settings → MCP Access** (see above), sent as `Authorization: Bearer fmcp_`. A `read`-scoped key may call only the read tools (`get_*`, `list_*`, `search`, `enter_project`, `retrieval_telemetry`); any write/delete tool is rejected with `403`. The allow-list is explicit rather than derived from the name — see `_READ_ONLY_TOOLS`, which is why the two reads without a read-shaped name are spelled out here. A `write`-scoped key may call everything. -### Claude Code (Project-scoped) +### Claude Code: the plugin (recommended) + +The plugin connects the MCP server and adds the hooks and skills that put Scribe's records in front of Claude as it works. Install it from this repo: + +``` +/plugin marketplace add https://git.fabledsword.com/bvandeusen/FabledScribe.git +/plugin install scribe@scribe-plugin +``` + +It asks for your Scribe base URL, an API key (kept in your OS keychain) and, optionally, a project id. Settings → MCP Access shows the exact command for your instance. [`plugin/README.md`](../plugin/README.md) describes each hook. + +### Claude Code without the plugin Add a `.mcp.json` at the project root. The server `type` is `http` and the URL is your instance's `/mcp` endpoint: @@ -78,21 +89,24 @@ specific project at a dev instance or an admin key. ### Available Tools -The tool surface is large (~70 tools) and evolves with the app, so the live -registration in **`src/scribe/mcp/tools/`** is the source of truth rather than a -table here. The tools are grouped by family: +There are well over a hundred tools and they change with the app, so the +registration in **`src/scribe/mcp/tools/`** is the reference. By family: | Family | Examples | Purpose | |--------|----------|---------| -| Notes | `create_note`, `get_note`, `update_note`, `delete_note`, `list_notes` | Free-form knowledge | -| Tasks | `create_task`, `update_task`, `add_task_log`, `start_planning` | Actionable work + plans | -| Projects / Milestones | `enter_project`, `get_project`, `create_milestone`, … | Containers and outcomes | -| Search / Recall | `search`, `get_recent`, `list_tags`, `retrieval_telemetry` | Semantic + structured recall, and the readout its thresholds are tuned from | -| Systems | `create_system`, `list_systems`, `list_system_records` | Reusable per-project subsystems/areas | -| Rulebooks | `list_rules`, `create_rule`, `create_project_rule`, `relate_rules`, … | Engineering/workflow rules | -| Processes | `list_processes`, `get_process`, `create_process` | Saved prompts/workflows | +| Notes | `create_note`, `get_note`, `update_note`, `create_records` | Knowledge, and several records that cite each other in one call | +| Tasks | `create_task`, `update_task`, `add_task_log` | Work, issues and spikes, and their logs | +| Projects / Milestones | `enter_project`, `start_planning`, `get_milestone` | Projects and their plans | +| Search / Recall | `search`, `what_might_apply`, `get_recent` | Recall by meaning, and the rules that may apply to an act | +| Systems | `create_system`, `list_system_records`, `map_system_to_canonical` | A project's areas | +| Rules | `create_rule`, `create_project_rule`, `relate_rules`, `rule_history`, `list_moments` | Rules, preferences, their relations and moments | +| Lessons | `create_lesson`, `judge_lesson_link` | Lessons and the rules they instance | +| Snippets / Shapes | `create_snippet`, `verify_snippet`, `classify_shapes`, `list_shapes` | Prior art and the shape ledger | +| Design systems | `resolve_design_system`, `get_design_system_stylesheet` | Tokens and guidance for UI work | +| Family canon | `list_family_ideas`, `assess_family_adoption` | Ideas shared across projects | +| Processes | `list_processes`, `get_process`, `create_process` | Stored procedures | +| Retrieval | `retrieval_telemetry`, `tune_retrieval` | What was surfaced, and the thresholds | | Trash | `list_trash`, `restore`, `purge_trash` | Recoverable deletes | -| Admin | `get_app_logs` (write/admin key) | Diagnostics | Server-level usage guidance — when to reach for each entity, the recall-before-acting reflex, and the rulebook conventions — is delivered to the diff --git a/docs/api-reference.md b/docs/api-reference.md index a5826b24..fa66ae11 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -52,6 +52,9 @@ endpoint at `/mcp`, not these REST routes. | POST | `/api/notes/:id/versions/:vid/pin` | Pin a version | | GET / PUT / DELETE | `/api/notes/:id/draft` | Unsaved-edit draft (restore across page loads) | | GET | `/api/notes/graph` | Knowledge-graph nodes/edges | +| GET | `/api/notes/duplicates` | Near-duplicate notes or tasks, grouped | +| GET | `/api/notes/due-for-verification` | Notes that carry a check, oldest verification first | +| POST | `/api/notes/:id/verify` | Record that a note's check was run `{still_true}` | ## Tasks @@ -86,7 +89,55 @@ endpoint at `/mcp`, not these REST routes. | GET / POST | `/api/projects/:id/systems` | List / create systems | | GET / PATCH / DELETE | `/api/projects/:id/systems/:sid` | Read / update / delete | | GET | `/api/projects/:id/systems/:sid/records` | Records linked to a system | +| PUT | `/api/systems/:sid/canonical` | Map a system to a shared area `{canonical_id}` (`null` unmaps) | +| GET / POST | `/api/canonical-systems` | List / create shared areas | +| PATCH | `/api/canonical-systems/:id` | Update a shared area | | GET | `/api/projects/:id/issues` | Project issues | +| PUT | `/api/projects/:id/design-system` | Point the project at a design system `{design_system_id}` (`null` clears) | +| GET / PUT | `/api/projects/:id/platforms` | The project's platforms `{platforms: {slug: "declared" \| "rejected" \| null}}` | +| PUT | `/api/projects/:id/forge` | Pin the project to one forge connection, or clear it | +| GET / POST | `/api/platforms` | Platform catalogue / add a platform | +| PATCH | `/api/platforms/:id` | Update a platform | + +## Snippets and lessons + +| Method | Path | Description | +|--------|------|-------------| +| GET / POST | `/api/snippets` | List / create snippets | +| GET / PATCH / DELETE | `/api/snippets/:id` | Read / update / delete | +| POST | `/api/snippets/:id/verify` | Record a drift-check verdict `{status, detail, path}` | +| GET | `/api/snippets/duplicates` | Near-duplicate snippets, grouped into merge candidates | +| POST | `/api/snippets/:id/merge` | Merge source snippets into this one | +| POST | `/api/snippets/:id/unmerge` | Pull one source back out `{source_id}` | +| GET / POST | `/api/lessons` | List / create lessons | +| GET / PATCH / DELETE | `/api/lessons/:id` | Read / update / delete (to the trash) | +| GET | `/api/lessons/rule-candidates` | Rules a draft lesson resembles, before it is saved | +| GET | `/api/lessons/taught-by/:record_id` | Lessons drawn from one record | +| PUT | `/api/lessons/:id/rules/:rule_id` | Confirm or reject a lesson → rule link | + +## Design systems + +| Method | Path | Description | +|--------|------|-------------| +| GET / POST | `/api/design-systems` | List / create | +| GET | `/api/design-systems/starter-roles` | Starter token roles for the creation form | +| GET / PATCH / DELETE | `/api/design-systems/:id` | Read / update / delete | +| GET | `/api/design-systems/:id/resolved` | The effective token set, inheritance applied | +| GET | `/api/design-systems/:id/stylesheet` | The generated stylesheet | +| GET | `/api/design-systems/:id/snippet-check` | Snippets that disagree with the system's sheet | +| GET / POST | `/api/design-systems/:id/tokens` | This system's own tokens / add one | +| PATCH / DELETE | `/api/design-tokens/:id` | Update / delete a token | + +## Family canon + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/family/ideas` | Family ideas | +| GET | `/api/family/ideas/:id` | One idea with its references | +| POST | `/api/family/ideas/:id/propose` · `/promote` · `/retire` | Move an idea through its lifecycle | +| GET | `/api/family/matrix` | Projects × canon ideas, with each project's answer | +| GET | `/api/family/decisions` | Promotion and retirement decisions | +| POST | `/api/family/decisions/:id/undo` | Undo a decision | ## Knowledge browse @@ -118,6 +169,27 @@ endpoint at `/mcp`, not these REST routes. | POST | `/api/rules/:id/move` | Move a rule: `{topic_id}` makes it global, `{project_id}` makes it that project's | | GET | `/api/projects/:id/rules` | A project's own rules, and the global rules tagged to its areas | | POST | `/api/projects/:id/rules` | Create a project-scoped rule | +| GET | `/api/rules/:id/versions` | A rule's edit history, newest first | +| GET | `/api/rules/:id/versions/:vid` | One earlier version in full | +| POST | `/api/rules/:id/versions/:vid/restore` | Put a preference back to an earlier version (preferences only) | +| GET | `/api/rules/drift` | Preferences that have been rewritten, most recent first | +| POST | `/api/rules/:id/relations` | Relate this rule to another (`co_surfaces`, `overrides`, `elaborates`) | +| DELETE | `/api/rule-relations/:id` | Remove a relation | +| GET | `/api/rules-due-for-verification` | Rules that carry a check, oldest verification first | +| POST | `/api/rules/:id/verify` | Record that a rule's check was run `{still_true}` | + +## Retrieval + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/retrieval/surfaces` | Every tunable surface with its threshold and budget | +| PUT | `/api/retrieval/surfaces/:surface` | Change one surface's setting, recorded with its reason | +| GET | `/api/retrieval/tuning-history` | Changes to retrieval on this install, newest first | +| GET | `/api/retrieval/moments` | The moments of work rules mount on, and the actions that reach each | +| POST / DELETE | `/api/retrieval/moments/mappings` | Map / unmap an action to a moment | +| GET | `/api/retrieval/moments/proposals` | Proposed rule → moment mounts awaiting a judgment | +| POST | `/api/retrieval/moments/proposals/judge` | Confirm or reject proposals | +| GET | `/api/retrieval/reply-shapes` | The default reply shapes and the moment each arrives at | ## Sharing @@ -154,6 +226,9 @@ endpoint at `/mcp`, not these REST routes. | GET / PUT | `/api/profile` | Read / update the per-user profile | | GET / PUT | `/api/settings` | Read / update key-value settings | | GET | `/api/settings/search` | SearXNG configuration status | +| GET / POST | `/api/settings/forge-connections` | Your git forge connections / add one | +| PUT / DELETE | `/api/settings/forge-connections/:id` | Update / remove a connection | +| POST | `/api/settings/forge-connections/:id/test` | Check that the forge is reachable and accepts the token | ## API keys @@ -164,10 +239,19 @@ endpoint at `/mcp`, not these REST routes. ## Plugin (Claude Code) +These serve the plugin's hooks. + | Method | Path | Description | |--------|------|-------------| -| GET | `/api/plugin/context` | SessionStart context payload (rules + active-project) | -| GET | `/api/plugin/retrieve` | Title-first knowledge-injection candidates | +| GET | `/api/plugin/context` | Session-start context: the active project and its state | +| GET | `/api/plugin/retrieve` | Records and rules matching the operator's prompt | +| GET | `/api/plugin/tool-rules` | Rules matching a tool call about to run | +| GET | `/api/plugin/moment-tools` | The tools whose calls can reach a moment | +| POST | `/api/plugin/moment` | The rules mounted on the moments a tool call reaches | +| POST | `/api/plugin/reply-rules` | Whether the reply ending a turn is held for one read | +| POST | `/api/plugin/rule-opened` | Records that a session opened a rule it was shown | +| GET | `/api/plugin/shape-check` | The end-of-turn question about what this turn wrote | +| GET | `/api/plugin/claim-session` · `/release-session` | Bind a session to its task claims / release them at session end | | GET | `/api/plugin/processes` | Stored Processes for skill-stub sync | | GET | `/api/plugin/prior-art` | Write-path hint for the plugin hooks (params: `path`, `code`, `repo`, `shapes`, `exclude_ids`, `exclude_sync_ids`, `exclude_derive`); returns `context`, `note_ids`, `sync_note_ids`, `stamped`, `divergence`, `derive`, `derive_keys` | | GET / POST | `/api/projects//coverage`, `…/coverage/refresh` | Shape-ledger accounting (`pattern_coverage` line, counts, `derive_groups` — css groups carry `consumers`, `derive_new`, `unused_css`, `divergence`, `recheck`) | @@ -197,6 +281,16 @@ endpoint at `/mcp`, not these REST routes. | GET / PUT | `/api/admin/base-url` | Get / set the public base URL | | GET / PUT | `/api/admin/db-maintenance` (+ `/health`, POST `/run`) | VACUUM schedule, health, manual run | | POST / GET / DELETE | `/api/admin/invitations` (+ `/:id`) | Create / list / revoke invite links | +| GET / PUT | `/api/admin/backups` | Scheduled backups: destination, schedule and recent runs | +| POST | `/api/admin/backups/check` | Ask the destination whether it would accept a backup | +| POST | `/api/admin/backups/run` | Start a backup now | +| GET / PUT | `/api/admin/forge-webhook` | The forge push-webhook settings | + +## Webhooks + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/webhooks/forge` | Push events from a git forge, verified with `FORGE_WEBHOOK_SECRET` **(public)** | ## Scribe MCP diff --git a/docs/architecture.md b/docs/architecture.md index 7329c0d8..71718522 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,380 +1,95 @@ # Architecture +Scribe is one app container and one PostgreSQL container. There is no language model in the stack: Claude works with Scribe from outside, through the MCP endpoint and the Claude Code plugin, and people use the web UI. + ## Stack -| Layer | Technology | Notes | -|-------|-----------|-------| -| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API | -| Editor | Tiptap (ProseMirror) with custom slash-command extension | | -| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files | -| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver | -| LLM | Ollama (local) | Any OpenAI-compatible API also works | -| Search | SearXNG (optional, self-hosted) | Web search + image search | -| Push | Web Push / VAPID (pywebpush 2.x) | | -| Deployment | Docker Compose | Single-container app + separate DB + LLM service | +| Layer | Technology | +|-------|-----------| +| Frontend | Vue 3, TypeScript, Vite, Pinia, Vue Router; Tiptap for the editor | +| Backend | Python 3.14, Quart (async), served by Hypercorn | +| MCP | The `mcp` SDK's server, mounted in the same app at `/mcp` (streamable HTTP) | +| Database | PostgreSQL 17 with pgvector; SQLAlchemy 2 (async, asyncpg); Alembic migrations | +| Embeddings | fastembed running in-process (`BAAI/bge-small-en-v1.5`, 384 dimensions) | +| Plugin | Claude Code hooks (POSIX shell + awk), skills and a slash command, shipped from this repo | -## High-Level Component Diagram +## How the pieces fit ``` -┌─────────────────────────────────────────────┐ -│ Docker Compose │ -│ │ -│ ┌──────────────────────┐ ┌────────────┐ │ -│ │ scribe │ │ ollama │ │ -│ │ ┌────────────────┐ │ │ │ │ -│ │ │ Quart Server │ │ │ LLM API │ │ -│ │ │ ┌──────────┐ │ │ │ │ │ -│ │ │ │ Vue SPA │ │ │ └────────────┘ │ -│ │ │ │ (static) │ │ │ ▲ │ -│ │ │ └──────────┘ │ │ │ │ -│ │ │ ┌──────────┐ │ │ HTTP/REST │ -│ │ │ │ /api/* │──┼──┼─────────┘ │ -│ │ │ └──────────┘ │ │ │ -│ │ │ │ │ │ ┌────────────┐ │ -│ │ │ ▼ │ │ │ PostgreSQL │ │ -│ │ │ ┌──────────┐ │ │ │ 16 │ │ -│ │ │ │ asyncpg │──┼──┼──▶ │ │ -│ │ │ └──────────┘ │ │ └────────────┘ │ -│ │ └────────────────┘ │ │ -│ └──────────────────────┘ │ -└─────────────────────────────────────────────┘ + Claude Code ──── plugin hooks ──── /api/plugin/* ┐ + │ │ + └────────── MCP tools ──────── /mcp ├── Quart app ── PostgreSQL + pgvector + │ │ + Browser ──────── Vue SPA ────────── /api/* ┘ └── in-process jobs and embedder ``` -## Project Structure +- **The web UI** is a Vue single-page app, built into the image and served by Quart. Every unknown path that isn't under `/api/` falls back to `index.html`; `/api/*` always answers JSON. +- **The REST API** (`src/scribe/routes/`) backs the UI, one blueprint per resource. +- **The MCP server** (`src/scribe/mcp/`) is how Claude reads and writes records. Each module in `mcp/tools/` registers its tools, and the server's instructions and each tool's docstring are the contract an agent reads. Requests authenticate with an API key as a Bearer token. +- **The plugin** (`plugin/`) runs in the user's Claude Code. Its hooks call `/api/plugin/*` to put the right context in front of Claude at the right moment: the project at session start, matching rules and prior art before a prompt or a tool call, the rules mounted on a moment of work, and a check on the reply that ends a turn. Its skills describe the working practices. See [API Keys & MCP](api-keys-and-mcp.md) to install it. +- **Services** (`src/scribe/services/`) hold the logic both doors share. Routes and MCP tools stay thin: parse the request, check access, call a service, return a plain dict. -``` -scribe/ -├── docker-compose.yml # Development stack -├── docker-compose.prod.yml # Production stack (Docker Swarm) -├── Dockerfile # Multi-stage build (Node → Python) -├── alembic/ # Database migrations -│ └── versions/ # Migration files (idempotent raw SQL) -├── src/scribe/ -│ ├── app.py # Quart app factory + blueprint registration -│ ├── config.py # Config class (reads env vars) -│ ├── auth.py # login_required decorator, session checks -│ ├── models/ # SQLAlchemy models -│ ├── mcp/ # In-app MCP server (FastMCP, mounted at /mcp) -│ │ ├── server.py # FastMCP instance + instructions + Quart mount -│ │ └── tools/ # Tool modules (notes, tasks, projects, rulebooks, …) -│ ├── routes/ # API blueprints (one file per resource) -│ ├── services/ # Business logic (access, llm, tools, sharing, …) -│ └── static/ # Built Vue SPA (generated at Docker build time) -└── frontend/ - └── src/ - ├── views/ # Page-level Vue components - ├── components/ # Reusable UI components - ├── composables/ # Vue composables (autosave, shortcuts, …) - ├── stores/ # Pinia stores (auth, chat, notes, notifications, …) - └── api/ # Typed API client (client.ts) -``` +## Data model -## Key Design Decisions +Most records are rows in one `notes` table, told apart by kind: -**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build. +- **Notes**, and the specialised kinds that are notes underneath: **snippets** (recorded prior art, with a canonical code location), **lessons** (transferable insights, linked to the rules they instance) and **processes** (stored procedures, synced to Claude Code as skills). +- **Tasks**: a note with a status. A task's kind is `work`, `issue` or `spike`. Tasks carry work logs, sub-tasks, recurrence and a session claim. -**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/` route intercepting API GETs. +Around them: -**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity. +- **Projects** group work and carry a goal. **Milestones** are plans: a design body plus ordered step tasks. +- **Systems** are a project's named areas; records are tagged with them, and each maps to a canonical system shared across projects. +- **Rulebooks → topics → rules.** A rule in a rulebook is global; a rule on a project applies there only. Rules have a trigger (`when_to_apply`), versions, relations to other rules, and can be mounted on **moments** of work (`work.change`, `reply.report`, ...), which plugin actions reach through moment mappings. +- **Family canon**: ideas shared across projects on the same platform, and each project's answer to them (adopted, variant, owed). +- **Design systems**: tokens and guidance, with inheritance; the stylesheet is generated from them. +- **The shape ledger**: code shapes extracted from bound repos and their classification against snippets. +- **Embeddings** for notes (chunked), rules, milestones and systems, in pgvector columns. +- **Telemetry**: retrieval logs, note and rule usage, and the tuning history of retrieval thresholds. +- **Users, groups, shares and API keys.** Access is decided in the app (`services/access.py`): owners, direct shares and group shares, with read or write permission. -**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching. +Deleted records go to the trash first and can be restored until retention purges them. -**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture. +## Retrieval -**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`. +Search and the plugin's context injection run on the same pipeline (`services/retrieval_pipeline.py`): embed the query, rank stored chunks by cosine similarity in pgvector, keep what clears the surface's threshold, and log what was shown. Each surface (MCP search, auto-inject, the pre-tool rule arm, the write-path prior-art arm, ...) is a spec over those stages with its own thresholds and budget. The thresholds are settings an admin can tune, and every change is recorded with its reason. When the embedder is unavailable, search falls back to keyword matching. -**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins. +The embedding model downloads into `FASTEMBED_CACHE_DIR` (default `/data/fastembed-cache`) on first use, so the `/data` volume keeps it across restarts. -**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend. +## Background work -**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses. +All in-process; there is no separate worker container. -**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var. +- At startup the app records any shipped default that moved in this release, starts its schedulers, and only then backfills missing embeddings, so the backfill never competes with startup for the database. +- Recurring tasks spawn every 15 minutes. Daily jobs: version auto-pinning (03:00 UTC), trash purge (03:30 UTC), targeted `VACUUM ANALYZE` (04:00 UTC by default). Scheduled backups check hourly whether one is due ([Backups](backups.md)). +- Log, notification and auth-token retention run on their own loops. +- A diagnostics heartbeat writes to `/data/diagnostics`. -**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy. - -**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs. - -## Data Model - -### Users - -| Column | Type | Notes | -|--------|------|-------| -| `id` | int PK | | -| `username` | text UNIQUE NOT NULL | | -| `email` | text nullable | | -| `password_hash` | text nullable | NULL for OAuth-only accounts | -| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier | -| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' | -| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions | -| `created_at` | timestamptz | | - -### Notes (unified — includes tasks) - -| Column | Type | Notes | -|--------|------|-------| -| `id` | int PK | | -| `title` | text | | -| `body` | text | Markdown | -| `tags` | ARRAY[text] | GIN indexed; explicitly set by client | -| `parent_id` | int FK self nullable | Sub-tasks / sub-notes | -| `user_id` | int FK users nullable | CASCADE | -| `project_id` | int FK projects nullable | | -| `milestone_id` | int FK milestones nullable | | -| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task | -| `priority` | text nullable | `none`/`low`/`medium`/`high` | -| `due_date` | date nullable | | -| `created_at`, `updated_at` | timestamptz | | - -Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`. - -### Settings - -Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `journal_config` (JSON: locations, temp_unit, prep schedule), `user_timezone`, `voice_*`, etc. - -### Conversations / Messages - -`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`journal`/`mcp`), `day_date` (YYYY-MM-DD for journal conversations), `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`. -`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`. - -Title auto-generated by LLM on first exchange, re-generated every 10th message. - -### Projects / Milestones - -`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps. -`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps. - -### Sharing & Access - -`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`. - -`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles. - -Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission. - -### Journal-Related Tables - -`weather_cache`: per-user, per-`location_key` cache. Columns: `user_id`, `location_key` (`home`/`work`/etc.), `location_label`, `forecast_json` (Open-Meteo response), `previous_json` (last forecast, used to detect changes), `fetched_at`. Lat/lon are *not* stored on the cache row — they live in the user's `journal_config.locations.{home|work}` setting and are used at refresh time. - -### API Keys - -`api_keys`: `id`, `user_id` FK CASCADE, `prefix` (first 8 chars, displayed in UI), `key_hash` (SHA-256 of full key — full key never stored), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at`. - -### App Logs - -`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB. - -## Detailed File Reference - -### Backend (`src/scribe/`) - -| File | Responsibility | -|------|---------------| -| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` | -| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper; accepts session cookie or `Authorization: Bearer ` | -| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods | -| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` | -| `models/note.py` | Unified Note model (notes + tasks) | -| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` | -| `models/conversation.py` | `Conversation` + `Message` models | -| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB | -| `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` | -| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) | -| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) | -| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards | -| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) | -| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm | -| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts | -| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) | -| `routes/task_logs.py` | Task work log CRUD (`GET/POST/DELETE /api/tasks/:id/logs`) | -| `routes/projects.py` | Projects CRUD + summary endpoint | -| `routes/milestones.py` | Milestones CRUD under `/api/projects/:id/milestones` | -| `routes/settings.py` | Per-user settings key-value (`GET/PUT /api/settings/:key`) | -| `routes/journal.py` | Journal config CRUD; today/day/days conversation accessors; weather endpoints (cached/current/refresh/geocode); moments CRUD; trigger-prep | -| `routes/groups.py` | Group CRUD + membership management (admin) | -| `routes/shares.py` | Share project/note with user or group; revoke; list incoming shares | -| `routes/in_app_notifications.py` | In-app notification list + mark-read + count | -| `routes/push.py` | Web Push subscription subscribe/unsubscribe; VAPID public key | -| `routes/users.py` | User profile; admin user list + delete | -| `routes/images.py` | Serve cached images at `/api/images/` | -| `routes/export.py` | `GET /api/export` — personal Markdown ZIP or JSON array download | -| `routes/api_keys.py` | API key CRUD (`GET/POST/DELETE /api/api-keys`) | -| `mcp/server.py` | Mounts the in-app FastMCP server at `/mcp` (streamable HTTP, Bearer auth) | -| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation | -| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search | -| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens | -| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` | -| `services/api_keys.py` | `generate_key()`, `create_api_key()`, `list_api_keys()`, `revoke_api_key()`, `lookup_key()` (SHA-256 hash lookup) | -| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard | -| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` | -| ~~`services/intent.py`~~ | Removed — intent routing eliminated; the main model handles all tool routing directly | -| `services/tools/` | LLM tool package — decorator-based registry (`_registry.py`), shared helpers (`_helpers.py`), and one module per domain: `notes.py` (create/update/delete/search/list/read), `tasks.py` (list/log_work), `entities.py` (save_person/save_place/lists), `projects.py`, `calendar.py`, `web.py`, `rag.py`, `profile.py`, `rss.py`, `weather.py`, `utility.py` (calculate). `execute_tool()` and `get_tools_for_user()` are the public API. | -| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) | -| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) | -| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) | -| `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management | -| `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata | -| `services/note_drafts.py` | Per-user per-note draft persistence (AI Assist pending state) | -| `services/settings.py` | `get_setting()`, `set_setting()`, `get_all_settings()` — key-value per user | -| `services/tag_suggestions.py` | `/api/notes/suggest-tags` — LLM-generated tag suggestions for note body | -| `services/access.py` | Permission resolution for all shared resources | -| `services/sharing.py` | Create/revoke shares; list shares; `get_shared_with_me()` | -| `services/groups.py` | Group CRUD; membership management | -| `services/notifications.py` | Create/read/mark-read in-app notifications | -| `services/task_logs.py` | Append/list/delete work log entries on tasks | -| `services/journal_prep.py` | Deterministic data gather (tasks/events/weather/projects/recent moments/open threads) → LLM prose opener; persisted as the first assistant message of today's journal Conversation | -| `services/journal_pipeline.py` | System-prompt builder for journal conversations; calls `build_profile_context()` so the LLM sees the user's profile + learned summary | -| `services/journal_scheduler.py` | APScheduler `BackgroundScheduler`; per-user prep job; live-reschedule via `update_user_schedule()`; catch-up logic for missed runs | -| `services/journal_search.py`, `services/moments.py` | Moment recording + search across journal history | -| `services/user_profile.py` | `build_profile_context()` consolidates profile + observations + learned_summary into a system-prompt block | -| `services/research.py` | SearXNG research pipeline: sub-queries → parallel fetch → outline → section synthesis → executive summary → index note with linked section notes | -| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools | -| `routes/events.py` | `/api/events` — event CRUD routes | -| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation | -| `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed | -| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap | -| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) | -| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup | -| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) | -| `services/email.py` | SMTP email sending for password reset; reads `SMTP_*` env vars | -| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache | -| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items | -| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection | - -### Frontend (`frontend/src/`) - -| File | Responsibility | -|------|---------------| -| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount | -| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect | -| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth | -| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` | -| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` | -| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` | -| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` | -| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` | -| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` | -| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response | -| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete | -| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver | -| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView | -| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage | -| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel | -| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management | -| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage | -| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave | -| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks | -| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" | -| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" | -| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` | -| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration | -| `components/ShareDialog.vue` | `` modal; user tab + group tab; current shares list | -| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over | -| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave | -| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft | -| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards | -| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm | -| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state | -| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard | -| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` | -| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` | -| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete | - -## Key Services - -| Service | Responsibility | -|---------|---------------| -| `services/access.py` | Permission resolution for all shared resources | -| `services/llm.py` | `build_context()`, RAG injection, history summarisation | -| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management | -| `services/tools/` | LLM tool implementations (38 tools across 11 modules); decorator-based registry | -| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` | -| `services/journal_prep.py` | Data gather → LLM prose opener; persisted into today's journal conversation | -| `services/journal_scheduler.py` | APScheduler integration, per-user prep job, catch-up for missed runs | -| `services/backup.py` | Full and per-user backup export/restore (version 2 format) | -| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache | -| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 | +Every database wait on startup has a deadline. Migrations wait for the database to accept connections (up to five minutes) before they run, so the app starts cleanly whatever order the containers come up in. ## Authentication -**Session cookies** — `HttpOnly`, `SameSite=Lax`, optionally `Secure` (`SECURE_COOKIES` env var). Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear. +- **Sessions**: `HttpOnly`, `SameSite=Lax` cookies, `Secure` when `SECURE_COOKIES=true`. Changing a password bumps the user's session version and signs out their other sessions. +- **API keys**: `Authorization: Bearer fmcp_...`, stored hashed. A `read` key may only call read endpoints and read MCP tools. +- **OIDC** single sign-on with PKCE, optionally replacing local logins ([SSO / OAuth](sso-oauth.md)). +- **Rate limits** on register, login and forgot-password, per client IP (`TRUST_PROXY_HEADERS` reads the IP from the proxy's headers). -**Bearer token / API keys** — `Authorization: Bearer ` accepted by `_check_auth()` as an alternative to session cookies. The raw key is SHA-256 hashed and looked up via `services/api_keys.py`. Keys with `scope=read` are rejected on non-safe methods (`POST`/`PATCH`/`DELETE`). Used by Fable MCP and any external API consumers. +Credentials Scribe stores for outside services (forge tokens, the backup destination's S3 secret) are sealed under a key derived from `SECRET_KEY`. -**Local auth + OIDC/OAuth (PKCE)** — `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user. +## Repository layout -See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions. - -## LLM Pipeline Internals - -### Tool Routing - -No separate intent router — the main model handles all tool routing directly via Ollama's structured tool-calling output. The model receives the full tool schema list and decides whether to call a tool or respond conversationally. Extended reasoning (`think=True`) is always on for qwen3-class models: content-based gating was tried but exposed tool-call template fragility on short tool-intent prompts. - -### Tool Loop - -Multi-round tool loop (max 5 rounds). All implementations in `services/tools/` (decorator-based registry); `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation. - -**Unified `create_note` tool** — creates both notes and tasks. Setting `status` (e.g. `"todo"`) creates a task; omitting it creates a knowledge note. All task fields (due_date, priority, milestone, parent_task, recurrence_rule) are available on the single tool. - -**Duplicate protection on `create_note`:** -1. Exact title match (case-insensitive) → hard block, redirect to `update_note` -2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block -3. Semantic content similarity (threshold 0.90, body ≥ 80 chars) → soft block with `requires_confirmation: true` - -**Project resolution** (`_helpers.resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55. - -### Context Window and Summarisation - -`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure. - -History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens. - -### Web Research Pipeline - -`services/research.py` implements a full autonomous research pipeline: -1. Intent model generates 5 focused sub-queries -2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter) -3. Up to 15 unique URLs fetched in parallel -4. LLM generates an outline (2–8 sections with title + focus) -5. Each section synthesised in parallel from relevant sources -6. Executive summary generated from all section content -7. Index note created with executive summary + links to section notes; section notes linked back via `parent_id` -8. Falls back to single-note synthesis if outline generation fails - -SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests. - -### Image Cache - -`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/`. The user's browser never contacts the original image host. - -Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB). - -## RAG Pipeline - -1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags. -2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each). -3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include). -4. Explicitly included notes delivered full-body. -5. `excluded_note_ids` prevents the current note from being injected as its own context. - -### RAG Scope (three-value system) - -`conversations.rag_project_id` controls which notes are eligible for retrieval: - -| Value | Behaviour | -|-------|-----------| -| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` | -| `-1` | All notes — opt-in to global search | -| Positive int | That project's notes only | - -`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths. - -**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation: -- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects. -- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively. - -**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup. - -Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task). +``` +├── alembic/versions/ database migrations +├── src/scribe/ +│ ├── app.py app factory, blueprints, startup and shutdown +│ ├── config.py environment configuration +│ ├── models/ SQLAlchemy models +│ ├── routes/ REST blueprints +│ ├── mcp/ MCP server and its tool modules +│ ├── services/ shared logic +│ └── restore.py `python -m scribe.restore` for a fresh instance +├── frontend/src/ views, components, stores, composables, api client +├── plugin/ Claude Code hooks, skills and commands +├── scripts/ CI checks and the plugin version minter +└── tests/ unit tests, plus `-m integration` against real Postgres +``` diff --git a/docs/configuration.md b/docs/configuration.md index 1bd69907..88dc65c4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,108 +1,100 @@ # Configuration -Configuration is via environment variables. The `docker-compose.yml` file sets defaults for local development; override them in a `.env` file (gitignored). +Scribe reads its configuration from environment variables. Most settings people change day to day (registration, SMTP, retrieval thresholds, backups, forge connections) live in **Settings** in the web UI instead. -## Environment Variables +Any variable marked *secret* can also be read from a file: set `_FILE` to the path, for example a Docker secret at `/run/secrets/scribe_key`. + +## Environment variables ### Core | Variable | Default | Description | |----------|---------|-------------| -| `DATABASE_URL` | `postgresql+asyncpg://fabled:fabled@db/scribe` | PostgreSQL async connection string | -| `SECRET_KEY` | `dev-secret-change-me` | Session signing key, and the key stored credentials are sealed under — **change this in production, then keep it** | -| `SECRET_KEY_FILE` | — | Path to a Docker secret file containing the key (alternative to `SECRET_KEY`) | -| `LOG_LEVEL` | `INFO` | Logging verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) | -| `SECURE_COOKIES` | `false` | Set `true` when running behind TLS | -| `BASE_URL` | — | Public URL (e.g. `https://notes.example.com`) — required for OIDC redirect URIs and email links | +| `DATABASE_URL` *(secret)* | `postgresql+asyncpg://scribe:scribe@localhost:5432/scribe` | PostgreSQL connection string, using the `asyncpg` driver | +| `SECRET_KEY` *(secret)* | `dev-secret-change-me` | Signs sessions and seals stored credentials. Set it before first use and keep it (see below) | +| `BASE_URL` | `http://localhost:5000` | Public URL, used in email links and OIDC redirects | +| `SECURE_COOKIES` | `false` | `true` when Scribe is served over HTTPS | +| `TRUST_PROXY_HEADERS` | `false` | `true` behind a reverse proxy, to read the client IP from `X-Forwarded-For` / `X-Real-IP` | +| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING` or `ERROR` | +| `LOG_RETENTION_DAYS` | `90` | Days of app logs to keep | +| `FASTEMBED_CACHE_DIR` | `/data/fastembed-cache` | Where the embedding model is stored after its first download | -### LLM / Ollama +### Sign-in | Variable | Default | Description | |----------|---------|-------------| -| `OLLAMA_URL` | `http://ollama:11434` | Ollama API base URL | -| `OLLAMA_MODEL` | `llama3.2` | Default LLM model (used as fallback; per-user setting overrides this) | -| `EMBEDDING_MODEL` | `nomic-embed-text` | Model used for semantic search / RAG embeddings | -| `OLLAMA_NUM_CTX` | `16384` | Context window size passed to Ollama for all generation calls | - -### Authentication / OIDC - -| Variable | Default | Description | -|----------|---------|-------------| -| `LOCAL_AUTH_ENABLED` | `true` | Set `false` to disable local username/password login (SSO-only mode) | -| `OIDC_ISSUER` | — | OIDC issuer URL (e.g. `https://auth.example.com/application/o/fabled/`) | +| `LOCAL_AUTH_ENABLED` | `true` | `false` turns off username/password login, leaving SSO only | +| `OIDC_ISSUER` | — | OIDC issuer URL | | `OIDC_CLIENT_ID` | — | OIDC client ID | -| `OIDC_CLIENT_SECRET` | — | OIDC client secret | -| `OIDC_CLIENT_SECRET_FILE` | — | Docker secret file alternative to `OIDC_CLIENT_SECRET` | -| `OIDC_SCOPES` | `openid profile email` | Space-separated OIDC scopes to request | +| `OIDC_CLIENT_SECRET` *(secret)* | — | OIDC client secret | +| `OIDC_SCOPES` | `openid profile email` | Scopes to request | -See [sso-oauth.md](sso-oauth.md) for provider-specific setup. +SSO turns on when the issuer, client ID and secret are all set. See [SSO / OAuth](sso-oauth.md). -### Security / Proxy +### Email + +Admins can set SMTP in **Settings**; these are the defaults until they do. + +| Variable | Default | +|----------|---------| +| `SMTP_HOST` | — | +| `SMTP_PORT` | `587` | +| `SMTP_USERNAME` | — | +| `SMTP_PASSWORD` *(secret)* | — | +| `SMTP_FROM_ADDRESS` | — | +| `SMTP_FROM_NAME` | `Fabled Scribe` | +| `SMTP_USE_TLS` | `true` | + +### Integrations | Variable | Default | Description | |----------|---------|-------------| -| `TRUST_PROXY_HEADERS` | `false` | Set `true` when behind a trusted reverse proxy to read real IP from `X-Forwarded-For` / `X-Real-IP` | +| `PLUGIN_MARKETPLACE_URL` | this repo's git URL | The git URL shown in the plugin install command under Settings → MCP Access. An admin can override it in Settings | +| `FORGE_KIND`, `FORGE_BASE_URL`, `FORGE_TOKEN` *(secret)* | — | A git forge connection for admins' projects, so a token can live in a Docker secret. Each user can add their own under Settings → Integrations; a stored connection for the same host wins | +| `FORGE_WEBHOOK_SECRET` *(secret)* | — | Verifies push webhooks from the forge | +| `SEARXNG_URL` | — | A SearXNG instance, listed as the web-search backend under Settings → Integrations | -### Web Search / Images +## Running it -| Variable | Default | Description | -|----------|---------|-------------| -| `SEARXNG_URL` | — | SearXNG base URL for web search and image search tools | -| `IMAGE_CACHE_DIR` | `/data/images` | Directory for cached search images | -| `IMAGE_MAX_BYTES` | `5242880` (5 MB) | Maximum size for cached images | +### Quick start -### Data / Logging - -| Variable | Default | Description | -|----------|---------|-------------| -| `LOG_RETENTION_DAYS` | `90` | Days to keep app logs before automatic pruning | -| `DATA_DIR` | `/data` | Root directory for persistent data (VAPID keys, backups) | - -## Docker Compose Setup - -### Development (`docker-compose.yml`) +[`docker-compose.quickstart.yml`](../docker-compose.quickstart.yml) runs the published image with its own database: ```bash -# Copy the example env file -cp .env.example .env -# Edit .env to set a real SECRET_KEY +SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(32))') \ + docker compose -f docker-compose.quickstart.yml up -d +``` -# Start the stack +Open `http://localhost:5000`. The first account registered becomes the admin; after that, registration stays closed until an admin opens it in Settings or sends an invite. + +### From source + +```bash +cp .env.example .env # then set SECRET_KEY docker compose up --build - -# App available at http://localhost:5000 ``` -The first user to register becomes admin. Registration auto-closes after that (re-enable from Settings → Users). +### Docker Swarm -### Production (`docker-compose.prod.yml`) - -The production compose file adds: -- Docker Secrets for `SECRET_KEY_FILE` and `DATABASE_URL_FILE` -- Network isolation (internal bridge for DB + Ollama; only the app is externally accessible) -- Health checks and resource limits +[`docker-compose.prod.yml`](../docker-compose.prod.yml) is a stack file: the app and database on a private overlay network, with `SECURE_COOKIES` and `TRUST_PROXY_HEADERS` on. Provide `DB_PASSWORD` and `SECRET_KEY` in the environment: ```bash -# Create Docker secrets -echo "$(python3 -c 'import secrets; print(secrets.token_hex(32))')" | docker secret create fabled_secret_key - -echo "postgresql+asyncpg://fabled:strongpassword@db/scribe" | docker secret create fabled_db_url - - -# Deploy -docker stack deploy -c docker-compose.prod.yml fabled +DB_PASSWORD=... SECRET_KEY=... docker stack deploy -c docker-compose.prod.yml scribe ``` -## Production Deployment +The shipped files leave out healthchecks and restart policies on purpose. The app waits for its database on startup, and the orchestrator's defaults restart it if it ever exits. -### Reverse Proxy (Required) +### Volumes -Fabled Assistant does **not** handle SSL/TLS. Run it behind a reverse proxy: +- **Database** (`pgdata`): everything Scribe stores. +- **App data** (`/data`): the embedding model cache and diagnostics. Losing it costs a re-download, nothing else. -- **Nginx**, **Traefik**, or **Caddy** in front of the app container -- Terminate TLS at the proxy; forward to port 5000 -- **Do not expose port 5000 directly to the internet** -- Rate-limit auth endpoints: ≤ 5 req/min per IP on `/api/auth/login` and `/api/auth/register` +Back up the database with **Settings → Data → Scheduled backups** ([Backups](backups.md)). + +## Behind a reverse proxy + +Scribe doesn't terminate TLS. Put it behind Nginx, Traefik or Caddy, forward to port 5000, and keep port 5000 off the internet. Set `SECURE_COOKIES=true`, `TRUST_PROXY_HEADERS=true` and `BASE_URL`. -Example Nginx location block: ```nginx location / { proxy_pass http://127.0.0.1:5000; @@ -111,40 +103,18 @@ location / { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; - # Required for SSE streaming + # MCP responses can stream. proxy_buffering off; proxy_read_timeout 600s; - proxy_send_timeout 600s; } ``` -If using `TRUST_PROXY_HEADERS=true`, ensure your proxy strips any client-supplied `X-Forwarded-For` headers before adding its own. +With `TRUST_PROXY_HEADERS=true`, make sure the proxy replaces any `X-Forwarded-For` the client sends. -### Security Checklist +## Security checklist -- **Strong `SECRET_KEY`** — generate with: - ```bash - python3 -c "import secrets; print(secrets.token_hex(32))" - ``` - Or use Docker Secrets via `SECRET_KEY_FILE`. -- **Keep `SECRET_KEY` once it is set.** Scribe seals the credentials it stores under a key derived from it: forge-connection tokens and the backup destination's S3 secret. A copy of the database alone can't read them. If the key changes (for example, a restore onto an instance with a different key), those credentials can no longer be opened. Settings marks each one and asks for it again, and nothing else breaks. -- **`SECURE_COOKIES=true`** — must be set when running behind TLS. -- **`BASE_URL`** — set to your public URL; required for OIDC redirects and email links. -- **Registration** — auto-closes after the first user (admin). Re-enable from Settings → Users or send invite links. -- **Session invalidation** — changing or resetting a password bumps `session_version`, evicting all other active sessions. Button also available in Settings → Account. -- **Keep Ollama on an internal network** — both compose files keep Ollama off the host network. Never expose the Ollama port publicly. -- **Default SECRET_KEY warning** — the app logs a `WARNING` on startup if the default dev key is in use. - -## Model Management - -Models are managed through Settings → General → Model Management in the web UI. You can: -- Pull new models by name (streams progress via SSE) -- View installed models with size and loaded/unloaded state -- Delete unused models - -The app auto-warms user-preferred models on startup (only models already installed — never auto-pulls). The embedding model (`nomic-embed-text`) is auto-pulled on startup if missing. - -Recommended models for 2× 8 GB GPU: -- **`qwen3:8b`** — strong reasoning + tools, fits in 8 GB, supports thinking mode -- **`qwen2.5:7b`** — fast, tool-capable, smaller context -- **`llama3.1:8b`** — reliable baseline, widely tested with tools +- **Set a strong `SECRET_KEY`, and keep it.** Forge tokens and the backup destination's S3 secret are sealed under a key derived from it, so a copy of the database alone can't read them. If the key changes (say, a restore onto an instance with a different key), Settings marks those credentials and asks for them again; nothing else breaks. The app logs a warning at startup while the default key is in use, and refuses to start with it when `SECURE_COOKIES=true`. +- **`SECURE_COOKIES=true`** whenever Scribe is served over HTTPS. +- **Registration** is open only for the first account. Invite people, or open it briefly from Settings. +- **Signing everyone else out**: changing a password ends that user's other sessions; Settings → Account has a button for it too. +- **API keys** are per user, shown once, and can be read-only. Revoke any you no longer use under Settings → MCP Access. diff --git a/docs/design-system.md b/docs/design-system.md index ab9f3737..d22ed87e 100644 --- a/docs/design-system.md +++ b/docs/design-system.md @@ -354,220 +354,13 @@ If using Tailwind, extend the theme with these tokens rather than relying on def --- -*Last updated: April 25, 2026. Iterate as the family of apps grows.* +# Scribe ---- +The design system Scribe renders with is stored in Scribe itself: **Scribe**, which inherits **FabledSword**, edited at `/design-systems`. `frontend/src/assets/theme.css` is generated from it, and CI rejects a `var()` the sheet doesn't define. Where this page and the stored system differ, the stored system is right. Scribe applies the baseline above with these choices: -# Scribe-specific decisions in progress - -> This section tracks decisions made while adapting the FabledSword baseline above for Scribe specifically. Items here are *in progress* — once they feel solid, they get folded into the main body of the document (either as Scribe-specific extensions in the per-app section, or as updates to the universal rules where Scribe's needs reveal a gap in the baseline). - -*Iteration started: 2026-04-26. Foundation pass shipped 2026-04-27 in `7a9a8b7` (palette, fonts, light mode, action tokens, hardcoded indigo cleanup, warm-gold deprecation). Surface phase shipped 2026-04-27 across `93a3beb` → `3c1ec40` (Lucide migration, Hybrid-rule button reclassification per surface, long-form line-height, two-weights-only). The system is now applied end-to-end; this section will fold into the main body once the result has had time to settle in real use.* - -## Decisions made so far - -### Accent footprint — Hybrid rule (not Strict) - -The doc baseline says the per-app accent only appears on wordmark, active nav, cursor, and text selection — never on action buttons. Scribe currently uses indigo on essentially every interactive surface (CTAs, scrollbars, glows, borders, focus rings). Hard-cutting to the doc baseline would lose too much identity in one swing. - -**Hybrid rule:** the accent reserves a slightly larger footprint than the doc baseline, but still much smaller than today. - -- **Accent (dusty violet) lives on:** wordmark; active nav; cursor and text selection in editor surfaces; tags/pills/wikilinks; in-progress task badge; focus rings; **brand-moment CTAs** — chat Send, "Create note" empty-state CTA, journal Send, "Start journaling" empty-state. -- **Moss (sage-green primary) lives on:** Save / Submit / Confirm in forms and modals; generic affirmative actions where the button just means "do this thing" with no brand pretense. -- **Bronze (secondary):** Cancel-but-not-destructive, alternative paths. -- **Oxblood (destructive):** Delete / Remove (paired with an icon). -- **Pewter ghost:** tertiary actions, "later", "skip", "see also". - -**Rule of thumb:** if the user is engaging with a *Scribe-feature moment* (sending a chat, opening a fresh note, jumping into the journal), accent. If they're just *operating the software* (saving an edit, confirming a dialog), Moss. - -### Light mode — warm parchment, matched aesthetic - -The doc is dark-only. Scribe today supports both light and dark, and we keep both. The light mode is derived to *match* the dark mode aesthetic rather than defaulting to system white-and-ink. - -- Page background: in the `#F5F1E8` warm cream family (specific values TBD) -- Cards: near-white but slightly tinted -- Text: deep ink `#14171A` (mirroring Obsidian) -- Accent: same dusty violet `#5B4A8A` (works on both themes) - -The metaphor stays consistent across themes: ink on aged paper (light) ↔ parchment text on graphite (dark). Light mode is *not* the system standard look. - -**Known downside:** warm parchment backgrounds can fight with embedded color content. Mitigation: code blocks get a slight cool wash in light mode specifically, to keep syntax highlighting readable. - -### Status and priority palette — extend the doc's semantic set - -The doc's semantic colors (Success / Warning / Error / Info / Destructive) are leaner than what Scribe needs for task management. Rather than running a parallel palette, Scribe extends the doc by mapping its status/priority tokens onto doc primitives where they fit and defining new app-level tokens for the rest. - -**Status (task lifecycle):** - -| Token | Color | Source | Logic | -|---|---|---|---| -| `status-todo` | Pewter `#3F4651` | doc | Neutral, "not started yet" | -| `status-in-progress` | dusty violet `#5B4A8A` | accent | Active = brand moment per Hybrid | -| `status-done` | Moss `#4A5D3F` | doc Success | Affirmative completion | -| `status-cancelled` | Ash `#9C9A92` | doc | Faded, "let go" | -| `status-paused` | Warning `#8B6F1E` | doc | Stalled, needs attention — replaces the old warm gold treatment | - -**Priority (loudness scale):** - -| Token | Color | Source | Logic | -|---|---|---|---| -| `priority-low` | Info `#3D5A6E` | doc | Cool, FYI — quietest end of the spectrum | -| `priority-medium` | Warning `#8B6F1E` | doc | Golden, mid-attention | -| `priority-high` | Error `#C04A1F` | doc | Terracotta, urgent | -| `priority-none` | Vellum/Ash | doc | No signal | - -The priority row reads as a clean cool→warm gradient (slate blue → golden brown → terracotta), which matches the semantic loudness — coherence the current ad-hoc palette doesn't have. - -**Other functional tokens:** - -| Token | Color | Logic | -|---|---|---| -| `wikilink` | dusty violet | Editorial brand moment per Hybrid | -| `overdue` | Error `#C04A1F` | Same as priority-high — overdue IS a priority signal | -| `toast-success` | Moss | doc semantic | -| `toast-error` | Error | doc semantic | -| `toast-info` | Info | doc semantic | -| `tag-bg` / `tag-text` | accent at 15% / accent | Per doc pill recipe | - -Each token gets a `*-bg` companion at low alpha (matching the existing pattern in `theme.css`). - -**Removed:** the warm gold accent (`--color-accent-warm: #b8860b`). Its two jobs split: - -- Dates and timestamps (knowledge cards, event details, chat) → use `text-secondary` instead. Dates are metadata, not a brand surface; muted is the correct register. -- Paused project status → use the new `status-paused` (Warning `#8B6F1E`) row above. Same golden-brown family, semantically aligned. - -### Typography — adopt the doc's stack and scale - -Adopt the doc's type stack and scale verbatim, with one deferred verification (long-form line-height in practice). - -- **Body font: Inter.** Replaces Scribe's current system-stack body font. Doc-defined; no Scribe-specific divergence. -- **Type scale:** as in the doc table — Display 40 / H1 32 / H2 24 / H3 18 / Body 15 / Body small 13 / Label 12 / Code 13 / Tiny 11. -- **Two weights only:** 400 regular, 500 medium. No 600/700 (reads heavy in dark mode and against the muted palette). -- **Family rules:** Fraunces at 18px+ only (Display, H1, H2). H3 and below = Inter. Code = JetBrains Mono. -- **Line height:** 1.5 body, 1.3 headings, **1.7 for long-form reading surfaces** (notes, journal entries). -- **Sentence case for everything**, except the Tiny micro-label style which gets uppercase + 0.08em letter-spacing. - -Mechanical rollout — value swaps in `theme.css` plus loading Inter and JetBrains Mono from Google Fonts (Fraunces is already loaded). - -### Light mode — concrete palette - -Fills in the warm-parchment direction picked earlier. Treat these as starting values; tune in practice. - -| Token | Hex | Role | -|---|---|---| -| Page bg | `#F5F1E8` | Warm cream — the "paper" | -| Card bg | `#FBF8F0` | Near-white, slightly warm — raised surfaces | -| Inset bg (inputs, code) | `#EFEAE0` | Slightly darker than page, "inset" feeling | -| Text primary | `#14171A` | Deep ink — Obsidian inverted | -| Text secondary | `#5A5852` | Warm mid-grey | -| Text muted | `#9A9890` | Warm light grey for hints/metadata | -| Border default | `#D9D6CE` | Warm light pewter, hairline weight | - -**Code-block exception:** in light mode specifically, code blocks use a slight cool wash (e.g. `#EBEDF0`) instead of the warm inset bg, so syntax highlighting reads cleanly. This is the mitigation for the "warm bg fights colored content" downside. - -The accent (`#5B4A8A` dusty violet), Moss, Bronze, Oxblood, and the semantic color set are **identical across themes** — only the surface and text palettes flip. - -### Chat-bubble codification — keep the Illuminated Transcript pattern - -The existing chat-bubble pattern (informally called "Illuminated Transcript") gets written into the design system as a documented chat component. Other apps in the family that add a chat surface inherit the pattern; Scribe's existing implementation continues to work with only color shifts. - -**User bubble (whisper):** -- Background: transparent -- Border: 0.5px Pewter (was: indigo-tinted) -- Text color: secondary (Vellum dark / `#5A5852` light) -- Right-aligned, rounded except bottom-right (subtle "from-me" tail) - -**Assistant bubble (lit):** -- Background: card surface (Iron dark / `#FBF8F0` light) -- Border: none on top/right/bottom; **2px solid accent (dusty violet) on left edge only** -- Box-shadow: accent-tinted glow + standard depth shadow (formula: `0 4px 28px rgba(, 0.14), 0 2px 8px rgba(0,0,0,0.4)` in dark; lower alphas in light) -- Text color: primary (Parchment dark / Obsidian-inverted light) -- Left-aligned, rounded except bottom-left - -The 2px-accent left edge is the "illumination" — like an illuminated capital in a manuscript. The shadow is the lift. Together they make the assistant bubble read as the *primary* voice, while the user bubble is the *margin note*. - -**Inline tool-call cards (`ToolCallCard`)** rendered inside an assistant bubble do NOT get their own border (per the border philosophy — the bubble already contains them). They use a slight surface tint to differentiate. - -### Iconography — adopt Lucide, enforce a scale - -Scribe currently hand-inlines SVG paths in 16+ Vue files, with 5 different stroke weights and 8+ different sizes. The visual style is already outline + rounded caps + `currentColor` stroke (matches the doc's intent), but there's no shared source and no scale discipline. - -**Migration policy:** - -1. **Install `lucide-vue-next`** as the icon source. Replace hand-inlined SVGs with imported components. Single source of truth. -2. **Strict size scale: 16px and 24px only.** Today's mix of 12/13/14/15/17/18/20 collapses to those two. 16 for inline-with-text and small affordances; 24 for nav and primary actions. -3. **Stroke weight per the doc: 1.5 at 24px, 1 at 16px.** Lighter than the current default of 2 — reads more refined, matches the muted palette philosophy. Overrides Lucide's default. -4. **Outline by default; filled only for active/selected state.** Introduces a new affordance Scribe doesn't currently use — bookmark/pin/star icons can switch outline → filled to indicate active state. Reserve filled style strictly for this. -5. **No emoji in chrome.** Replace the 3 files' emoji usage in UI labels/buttons/badges/empty states with Lucide equivalents. Emoji remain fine in *user content* (note bodies, chat messages the user typed). - -Work cost: ~30-60 individual icon swaps across the 16 files. Mechanical; doesn't require redesign of any component. - -### Voice and tone — adopt principles, defer formal audit - -The doc's voice register applies to Scribe (understated mythic — plain for functional UI, flavored for empty/error/loading states). No formal sweep of every UI string yet. - -**Approach:** apply the voice opportunistically as components are touched in the polish pass — when redesigning a settings tab, an empty state, or an error toast, rewrite the copy at the same time using the doc's register and examples table as the guide. A standalone audit pass is deferred unless drift becomes visible. - -### Border philosophy — structural, not decorative - -The doc treats borders as *structural* (Pewter neutral hairlines that say "boundary"), not decorative (Scribe today uses indigo-tinted borders that say "branded edge"). That principle suggests removing borders in places where surface tint and spacing already communicate separation. - -**Borders to remove:** - -- List rows (NotesListView, TasksListView, conversation history) — surface contrast + spacing should separate rows; current border reads as "boxed-in" -- Inline `ToolCallCard` inside chat bubbles — the bubble is already a container; an extra border feels like double-wrapping -- Filter chips and search-bar pills with a background tint — background does the work -- Empty-state callouts with dashed/bordered "nothing here yet" boxes — tinted background reads cleaner - -**Borders to keep (genuinely structural):** - -- Standalone card containers (Notes viewer, Task viewer, the new daily prep card) -- Modal / dialog edges -- Code blocks (separates content type, not just space) -- Focus rings (accessibility) -- Major section dividers within a panel - -Border weight is not load-bearing for Scribe — happy to use the doc's 0.5px hairline default; the *placement* discipline matters more than the weight. - -## Open threads (next iterations) - -### Foundation pass — shipped 2026-04-27 (`7a9a8b7`) - -Mechanical token + font + light-mode rewrite of `frontend/src/assets/theme.css`, plus a sweep of hardcoded indigo and `--color-accent-warm` references across ~14 component files. Action tokens (`--color-action-primary` Moss, `--color-action-secondary` Bronze, `--color-action-destructive` Oxblood, `--color-action-ghost-border` Pewter) are defined but not yet applied — buttons still flow through `--color-primary` and read as dusty-violet gradients in the meantime, by design. Spec lives at `docs/superpowers/specs/2026-04-27-design-system-polish-foundation-design.md` (gitignored, local-only). - -### Surface phase — shipped 2026-04-27 (`93a3beb` → `3c1ec40`) - -Bundled as Hybrid (option C from the brainstorm): Lucide cross-cutting first, then surface-by-surface for the judgment work. Spec lives at `docs/superpowers/specs/2026-04-27-design-system-polish-surface-design.md` (gitignored, local-only). Seven PRs landed on `dev`: - -| PR | Commit | Surface | Notes | -|---|---|---|---| -| 1 | `93a3beb` | Lucide cross-cutting | 60 hand-inlined SVGs across 15 files → `lucide-vue-next`. Every chrome icon at 16 or 24. Emoji-as-icons (`✕`, `✓`, `🎤`, `📎`, `↻`, `↑`, `×`, `☀`/`☾`) swept across the chrome. AppLogo wordmark and the GraphView D3 mount kept as the legitimate exceptions. | -| 2 | `3d916d7` | Journal | Buttons audited; `↻` weather refresh → Lucide `RotateCcw`; assistant bubble line-height bumped; redundant `.journal-title` font-family dropped; dead `.news-section` CSS removed. | -| 3 | `4192a64` | Chat | `.message-content` long-form 1.7 line-height on assistant bubbles; ToolCallCard outer border removed (bubble already contains it; error state moved to a left-edge accent); bulk-delete recolored to Oxblood with Trash2 icon; `.bulk-link` accent → muted text. | -| 4 | `efb3534` | Knowledge cluster | `.prose` line-height bumped to 1.7 globally so all reading surfaces inherit. Save → Moss; Delete → Oxblood + Trash2; Edit / Advance → Moss; Convert / Share → Bronze. | -| 5 | `ff498ce` | Project + Workspace | Project save panel → Moss; milestone confirm/cancel → Moss/Bronze; modal-btn-danger and per-row delete affordances → Oxblood; "Open Workspace" stays accent (project-surface brand moment). | -| 6 | `541e2ed` | Settings | Densest button surface — every `.btn-save`, `.btn-primary`, `.btn-secondary`, `.btn-danger`, `.btn-danger-outline`, `.btn-toggle-open` reclassified per Hybrid; missing `.btn-danger` style block added (was unstyled before). | -| 7 | `3c1ec40` | Edge surfaces | Calendar `New Event` → Moss; EventSlideOver Save/Cancel/Delete reclassified; HomeView hero CTA stays accent (brand moment); two-weights-only sweep across Header/Home/Calendar/Graph. | - -**Cross-cutting changes folded in as the work touched files:** - -- Long-form 1.7 line-height on `.prose` (PR 4) — applies to Note viewer, Task viewer, chat assistant bubbles, anywhere markdown renders into a reading surface. -- Two-weights-only (400 + 500) — every `font-weight: 600` and `700` snapped to `500` across all surface PRs. -- Hardcoded `--color-danger` in destructive button contexts → `--color-action-destructive` (Oxblood). `--color-danger` (Error terracotta) preserved for validation/error messages, per the doc's distinction between Error and Destructive. -- Adjacent `×` / `✕` / unicode-arrow emoji swept opportunistically as files were touched (PRs 5, 7). - -### Out of scope — deferred indefinitely - -Items deliberately not addressed in this round; revisit when a real need surfaces: - -- Lucide stroke-weight overrides (doc spec: 1.5 at 24, 1 at 16; current: Lucide default 2). Touched components if they read too heavy in practice. -- Filled-as-active icon state — no current affordance uses it; introduce when bookmark/pin/star toggles are added. -- Type-scale / spacing-scale CSS variables — components keep literal values. -- Token rename to `--fs-*` namespace — Scribe is the only FabledSword app sharing this codebase. -- FabledSword lockup placement — waiting on the actual heraldic mark to be drawn. -- Standalone voice/tone audit across every UI string — opportunistic-only; full sweep deferred unless drift becomes visible. -- A handful of editor utility buttons (`.btn-suggest-tags`, `.btn-link-all`, AI assist generate/proofread/accept/reject set, etc.) — currently ghost-styled and visually compliant; revisited only if they read off in practice. - -### Open threads - -*New threads will accumulate here as gaps surface in real use.* +- **Accent footprint.** Dusty violet marks Scribe's own moments: the wordmark, active navigation, the editor's cursor and selection, tags and wikilinks, in-progress status, focus rings, and the call to action that starts something new (a first note, say). Operating the software uses the action colours: Moss to save or confirm, Bronze for alternatives, Oxblood with an icon to delete, Pewter ghost for tertiary actions. +- **Light and dark.** Both are supported. Light mode is warm parchment (`#F5F1E8` page, `#FBF8F0` cards, `#14171A` text) rather than white, with a cool wash on code blocks so syntax colours stay readable. The accent, action and semantic colours are the same in both. +- **Status and priority.** Status: todo Pewter, in progress the accent, done Moss, cancelled Ash, paused Warning. Priority runs cool to warm: low Info, medium Warning, high Error. Overdue uses Error. Each has a low-alpha `-bg` companion. +- **Type.** Inter for UI and body, Fraunces at 18px and up (display, H1, H2), JetBrains Mono for code. Two weights only, 400 and 500. Line height 1.5 for body, 1.3 for headings, 1.7 on reading surfaces such as a note. +- **Icons.** Lucide, at 16px inline and 24px for navigation and primary actions; no emoji in the interface. +- **Borders are structural.** Cards, dialogs, code blocks, focus rings and major dividers get borders; list rows, chips and empty states are separated by surface tint and spacing instead. diff --git a/docs/development.md b/docs/development.md index 2afdebe8..5777d238 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,179 +1,87 @@ # Development -## Workflow - -All development is Docker-based. Do not install Python or Node dependencies locally. +## Running it ```bash -# Start the full stack (app + PostgreSQL + Ollama) -docker compose up --build +docker compose up --build # app + PostgreSQL at http://localhost:5000 +docker compose up --build app # rebuild after a change +docker compose down -v # stop and wipe the database +``` -# Rebuild after backend changes (frontend changes require rebuild too) -docker compose up --build app +The image builds the frontend, so a frontend change needs a rebuild too. For quicker frontend work, run Vite against the running app; it proxies `/api` to port 5000: -# Reset everything (wipes database) -docker compose down -v && docker compose up --build +```bash +cd frontend && npm ci && npm run dev # http://localhost:5173 +``` -# Run checks (lint, format, typecheck, tests) -make check +## Checks -# Individual checks +CI runs these on every push; locally they need `uv sync --extra dev` and `cd frontend && npm ci`. + +```bash make lint # ruff check src/ -make fmt # ruff format src/ make typecheck # vue-tsc --noEmit make test # pytest tests/ +make check # all three ``` -## Frontend Hot Reload +Unit tests run without a database. Tests marked `integration` need a real PostgreSQL and run in CI's integration job, against a schema built by the real migrations. -The Docker setup does not include Vite's hot-reload dev server. After frontend changes, rebuild the image. For faster iteration during active frontend work, you can run Vite locally: +## Database migrations + +The container runs `alembic upgrade head` on every start, after waiting for the database to accept connections. ```bash -cd frontend -npm install -npm run dev # Vite dev server at http://localhost:5173 +docker compose exec app alembic revision -m "short_description" ``` -Point the Vite dev server at the backend by setting `VITE_API_BASE_URL=http://localhost:5000` (or configure `vite.config.ts` proxy). +- Number revisions sequentially (`0123_...`). Revision ids must fit in 32 characters, the width of `alembic_version.version_num`. +- Write both `upgrade()` and `downgrade()`. +- Adding a value to a `CHECK`-constrained column means dropping and re-adding the constraint in the same migration. +- Keep migrations to schema changes. Filling a derived table from a large one belongs in a background backfill after startup, where its cost doesn't hold up the boot. +- A migration that depends on application code (a key derivation, a serialiser) copies what it needs rather than importing it, so a later edit can't change what an old migration does. -## Database Migrations +## The plugin -Alembic migrations run automatically on container startup (`alembic upgrade head` in the `CMD`). +`plugin/` ships straight from git: installs fetch it through `.claude-plugin/marketplace.json`, and there is no build step. After changing anything under `plugin/` or `.claude-plugin/`, run `make mint-plugin` before committing. Claude Code only refreshes an installed plugin when its version changes, and CI fails a plugin change that didn't bump it. -To create a new migration: -```bash -# Inside the running app container -docker compose exec app alembic revision -m "description_of_change" -# Edit the generated file in alembic/versions/ -``` +The hooks must run on a bare machine: POSIX shell, awk, sed and the usual coreutils, plus `git` and `curl`. They fail open, so a slow or unreachable Scribe never blocks Claude Code. -Migration conventions: -- Use raw SQL with `IF NOT EXISTS` guards for idempotency -- Use `DO $$ BEGIN CREATE TYPE … EXCEPTION WHEN duplicate_object THEN NULL; END $$` for enum types -- Number migrations sequentially (e.g. `0027_add_something.py`) -- Always provide both `upgrade()` and `downgrade()` +## CI and releases -## CI/CD +CI runs on Forgejo Actions in the shared `ci-python:3.14` image ([`ci-requirements.md`](../ci-requirements.md) lists what it relies on). Six jobs: TypeScript typecheck, plugin hooks, Python lint (plus design-token and dangling-style checks), unit tests, integration tests, and the image build, which waits on all of them except the plugin job. -### Pipeline +| Push | Image tags | +|------|-----------| +| `dev` | `:dev`, `:` | +| `main` | `:latest`, `:` | +| a `v*` tag | `:latest`, `:`, `:` | -CI runs on Forgejo Actions, consuming the shared -[`ci-python:3.14`](https://git.fabledsword.com/bvandeusen/CI-runner) image -via `container.image` (Python 3.14 + Node 24 + ruff + uv + Docker CLI): +`:` never moves, so it is the rollback point. Images publish to `git.fabledsword.com/bvandeusen/fabledscribe`, with a registry layer cache at `:cache`. The build needs the `REGISTRY_USER` and `REGISTRY_TOKEN` (a token with `write:packages`) repository secrets. -| Trigger | Jobs | Docker tags pushed | -|---------|------|--------------------| -| Push to `dev` | typecheck + lint + test → build | `:dev`, `:` | -| Tag `v*` on `main` | typecheck + lint + test → build | `:latest`, `:`, `:` | -| Push to `main` | typecheck + lint + test | (no build) | +Work happens on `dev`. A release merges `dev` into `main` with a plain merge commit; a dated `vYYYY.MM.DD.HHMM` tag, matching the version the image reports, can mark it. -### Release Process - -1. Work on `dev` branch — CI validates on every push -2. When ready, open a PR from `dev` → `main` in Forgejo -3. Merge the PR -4. Create a release via the Forgejo UI on `main` with a `v*` tag (e.g. `v26.03.23.1` — CalVer: `YY.MM.DD.N`) -5. The tag push triggers CI → build job pushes `:latest` + `:` Docker images -6. After merging to main, sync dev back: - ```bash - git checkout dev && git merge main && git push origin dev - ``` - -### Runner - -CI jobs schedule against the `python-ci` runner label and run inside the -shared `git.fabledsword.com/bvandeusen/ci-python:3.14` image (see -`ci-requirements.md` for what this project relies on from the image). -The runner deployment lives outside this repo; image bumps happen in -[CI-Runner](https://git.fabledsword.com/bvandeusen/CI-runner) via Renovate. - -`infra/runner-compose.yml` + `infra/act-runner-config.yml` document the -runner-host deployment shape; the source of truth is the deployed -config on the runner host. - -### Docker Registry - -Images pushed to: `git.fabledsword.com/bvandeusen/scribe` -Cache tag: `:cache` (reduces build time ~80%) - -Required secrets (repo → Settings → Secrets → Actions): -- `REGISTRY_USER` — Forgejo username -- `REGISTRY_TOKEN` — Forgejo PAT with `write:packages` scope - -## Migration Chain - -Current migration sequence (all idempotent raw SQL): - -``` -0001 create_notes_table -0002 create_tasks_table -0003 task_note_companion (data migration) -0004 merge_tasks_into_notes -0005 add_chat_tables -0006 add_settings_table -0007 add_title_and_updated_at_indexes -0008 add_users_and_user_id -0009 add_message_status -0010 add_app_logs_table -0011 add_password_reset_tokens -0012 add_invitation_tokens -0013 add_tool_calls_to_messages -0014 add_note_embeddings -0015 add_oauth_fields -0016 add_image_cache -0017 add_projects -0018 add_push_subscriptions -0019 add_events (dead code — internal CalDAV/Radicale table; Radicale was removed) -0020 add_milestones -0021 add_task_logs -0022 add_note_versions_and_drafts -0023 add_tags_to_note_versions -0024 add_session_version -0025 add_sharing_and_notifications -0026 add_briefing_tables -0027 add_api_keys -``` - -**Important:** Do NOT use `op.create_table()` or `sa.Enum()` — SQLAlchemy's event system can fire `CREATE TYPE` even with `create_type=False`, causing failures on re-run. Always use raw SQL with `IF NOT EXISTS` / `DO $$ BEGIN ... EXCEPTION WHEN duplicate_object` guards. - -## Project Conventions +## Conventions ### Backend -- Services: `async with async_session() as session:` — import from `scribe.models` -- No `scribe.database` module -- Blueprint per resource: `routes/notes.py`, `routes/tasks.py`, etc. -- All business logic in `services/`; routes are thin wrappers -- Permission checks via `services/access.py` — never inline ownership checks in routes +- Services open their own session: `async with async_session() as session:` (from `scribe.models`). +- Routes and MCP tools stay thin; the logic lives in `services/`, shared by both. +- Access checks go through `services/access.py`, never inline ownership checks. +- Every wait on something outside the process has a deadline. ### Frontend -- API calls via `frontend/src/api/client.ts` typed helpers (`apiGet`, `apiPost`, `apiPatch`, `apiDelete`) -- Pinia stores for shared state; local `ref()` for component-only state -- Composables in `composables/` for reusable behaviour (autosave, keyboard nav, tag suggestions, …) -- Views are page-level components in `views/`; reusable UI in `components/` +- API calls go through the typed helpers in `frontend/src/api/client.ts`. +- Pinia stores for shared state, `ref()` for state local to a component; composables for reusable behaviour. +- Colours, spacing and type come from the design system's tokens in `theme.css`, which is generated, so CI rejects an unknown `var()`. -### Commit Style +### Commits ``` -type(scope): short description +type(scope): what the change does -Longer body if needed. - -Co-Authored-By: Claude Sonnet 4.6 +Why, and anything a reader tracing this change later needs. ``` -Types: `feat`, `fix`, `refactor`, `docs`, `chore`, `test` -Scopes: feature area (e.g. `chat`, `journal`, `mcp`, `notes`) - -## Testing - -```bash -# Run all tests -make test - -# Run specific test file -docker compose exec app /opt/venv/bin/pytest tests/test_auth.py -v -``` - -Tests are in `tests/`. They run against a real PostgreSQL instance in CI (not mocked). Keep tests integration-style where possible — mock failures have historically masked real migration bugs. +Types: `feat`, `fix`, `refactor`, `docs`, `chore`, `test`. Comments in the code describe how it works today; the history of a change belongs in its commit message. diff --git a/docs/features.md b/docs/features.md index 0ef84740..64d73114 100644 --- a/docs/features.md +++ b/docs/features.md @@ -1,181 +1,131 @@ # Features -Scribe is a self-hosted work system-of-record for software projects, built to be -driven by Claude Code. There is **no in-app LLM** — Claude is the sole assistant, -reaching Scribe through a built-in MCP endpoint and a bundled Claude Code plugin. The -web UI is a clean surface for humans to read and edit the same data. +Scribe is a self-hosted system of record for the work you do with Claude Code. Claude records what it learned, decided and built, and that record reaches the next session, so a solution is recalled rather than reinvented. There is no language model inside Scribe: Claude reaches it through the MCP endpoint and the Claude Code plugin, and the web UI is where people read and edit the same records. ## Notes -Write in Markdown with a live-preview editor (Tiptap/ProseMirror). Headings, bold, -italic, lists, code blocks, and task checklists render inline. A slash-command menu -(`/`) inserts common blocks. +A Markdown editor (Tiptap) with headings, lists, code blocks and checklists inline, and a `/` menu for blocks. -- **Wikilinks** — Link notes with `[[Title]]` or `[[Title|Display Text]]`. Clicking - navigates to (or auto-creates) the referenced note; the editor suggests existing - titles while you type `[[`. Backlinks appear in the note viewer sidebar. -- **Tags** — First-class `ARRAY[text]` column with autocomplete. Hierarchical tags - (`area/backend`) supported — filtering by `area` matches all `area/*` children. -- **Version history** — Every body edit snapshots a version (up to 20 per note). - Browse, diff, and restore from the editor's History panel. -- **Draft recovery** — In-progress edits persist across page loads and are restored - when you reopen a note. -- **Convert freely** — Turn a note into a task (sets `status=todo`) or back again. +- **Wikilinks**: `[[Title]]` or `[[Title|text]]`, suggested as you type; backlinks show in the viewer. +- **Tags**, with autocomplete. Hierarchical tags (`area/backend`) match their parent's filter. +- **Version history**: browse, diff and restore earlier versions of a note. +- **Draft recovery**: unsaved edits survive a reload. -## Tasks and Issues +## Tasks -Tasks carry status (`todo` → `in_progress` → `done`/`cancelled`), priority -(`none`/`low`/`medium`/`high`), due date, milestone assignment, and a parent task -(sub-tasks). Notes and tasks share one model — a task is a note with a status. +A task is a note with a status (`todo`, `in_progress`, `done`, `cancelled`), plus priority, due date, milestone and parent task. -- **Work logs** — Append timestamped progress entries (with optional duration) to a - task without rewriting its body; shown chronologically in the task view. -- **Issues** — A task whose `kind` is corrective: a problem you fixed or are fixing, - with symptom → root cause → fix in the body. An issue can link the task it arose - from and the System(s) it touches. -- **Recurring tasks** — An interval or calendar recurrence rule spawns the next - occurrence when a task is completed (a background job drains due spawns). -- **Sub-tasks** — Any task can have children via `parent_id`; the viewer shows them - inline. +- **Kinds**: `work`; `issue`, for corrective work, which can link the task it arose from; and `spike`, a time-boxed investigation whose output is an answer. +- **Work logs**: timestamped entries that record progress without rewriting the body. They are what the next session reads first. +- **Claims**: a session that logs on a task holds it, so after a context compaction the work comes back to that session, and other sessions can see who is on it. +- **Recurring tasks** spawn their next occurrence on a schedule. -## Projects and Milestones +## Projects, milestones and systems -- **Projects** — Group related notes and tasks. Title, description, goal, status - (`active`/`paused`/`completed`/`archived`), and a colour. -- **Milestones** — Ordered stages within a project. A milestone is also the home of a - **plan** — its body holds the design (Goal/Approach/Verification) and its child - tasks are the steps. Completion percentage is shown on the project page. - Milestones are searchable by meaning (`search(content_type="milestone")`), and an - agent starting a plan (`start_planning`, `create_milestone`) is handed the active - milestone that already has its title or reads as the same plan, so steps are added - there rather than to a parallel plan. The match threshold is in Settings. -- **Kanban view** — `/projects/:id` groups tasks by milestone in a column layout with - status-advance buttons on the cards. +- **Projects** carry a goal, a status and a colour, and group everything else. Entering a project gives Claude a short primer: its goal, recent plans and their next steps, open tasks, its systems and its design system. +- **Milestones are plans**: a body holding the design (goal, approach, verification) and ordered step tasks. Starting a plan that matches an active milestone hands back that milestone instead of opening a second one. +- **Systems** are a project's named areas ("auth", "retrieval"). Records are tagged with them, each has a charter saying what belongs there, and each maps to a canonical area shared across projects. +- **Duplicate gates**: creating a record that reads like an existing one returns the existing one to update instead. -## Systems +## Rules, lessons and moments -A **System** is a per-project, reusable, self-describing subsystem or area (e.g. -"auth", "billing"). Associate any note, task, or issue with a System so research, -build-work, and fixes for the same area line up and recurring problem-spots surface. +- **Rulebooks → topics → rules.** A rule in a rulebook applies in every project; a project rule applies only there. A project that departs from a global rule says so with an `overrides` relation. +- **Rules arrive by retrieval.** Nothing is preloaded: a rule reaches a session when what Claude is about to do matches its trigger (`when_to_apply`), whether that's the operator's message, a command or a file being written. +- **Moments**: a rule about *when* something is done (finishing a task, reporting back, pushing) can be mounted on a moment of work, and arrives whenever that moment happens, whatever the words look like. Settings → General shows which actions reach each moment. +- **Versions and verification**: every edit to a rule keeps the text it replaced. A rule that asserts a fact about someone else's software can say how to check it and when it stops being true, and a sweep lists rules due for a check. +- **Preferences** are rules with the force of guidance rather than obligation. +- **Lessons** record something learned the hard way, keyed to the situation where it applies, and link to the rules they are instances of. +- **Reply shapes**: the default shape of Claude's replies (completion report, plan for review, question) arrives at the moment it applies. Settings → General shows them. -## Rules and Rulebooks +## Snippets and the shape ledger -Scribe stores the operator's engineering and workflow **rules** so Claude follows them -across sessions. +- **Snippets** are recorded prior art: a reusable piece of code with when to reach for it, its signature and its canonical location in a repo. Claude searches them before building and is shown matching ones before it writes a file. +- **Forge integration**: with a git forge connected, Scribe checks a snippet's cached body against its source and marks it current or diverged. Push webhooks keep that fresh. +- **The shape ledger**: Scribe extracts the code shapes from a bound repo and tracks how each is accounted for: an instance of a snippet, a deliberate variant with its reason, or a one-off. Coverage shows what is still unclassified. -- **Rulebooks → topics → rules** — Rules are grouped by topic inside a rulebook. -- **Rules arrive by retrieval** — Nothing is preloaded. A rule reaches a - session when what the agent is about to do matches its trigger: a command, - a file being written, or the operator's own message. `when_to_apply` is - therefore the field that decides whether a rule is ever seen. -- **Global or project scope** — A rule in a rulebook is global: it applies in - every project. A project rule applies to that project only. Retrieval honours - the difference, so a session sees global rules plus its own project's, never - another project's. A project that departs from a global rule writes its own - and links it with an `overrides` relation. +## Family canon -## Stored Processes +Projects on the same platform (Android apps, Go services, ...) share good ideas. An idea promoted to the family's canon reaches every project on that platform, and each answers it: adopted, a variant with its reason, or owed (filed as a task). `/family` shows the ideas and every project's answers. -Reusable saved prompts (a note with `note_type=process`) — e.g. a drift-audit or a -DRY pass. The bundled plugin syncs each Process into a local Claude Code skill stub -(`/scribe:sync`) that auto-surfaces by relevance and fetches the live procedure on -demand. +## Design systems -## Search and Knowledge Injection +Tokens (colour, type, space, motion, ...) and written guidance, with inheritance: a project's design system can extend a house style and record where it departs. Scribe generates a stylesheet from it, checks code snippets against it, and gives Claude the values to use before it writes UI. `/design-systems` edits them. -- **Semantic search** — pgvector-backed similarity search over notes and tasks - (in-process `fastembed` embeddings; no external model). -- **Proactive knowledge-injection** — the plugin's `UserPromptSubmit` hook surfaces a - short, high-confidence menu of maybe-relevant note *titles* into Claude's context - each turn; Claude pulls a full body only when it judges it relevant. Gated so it - stays quiet on most turns and never repeats within a session. +## Stored processes -## Knowledge Graph +Saved procedures, such as a drift audit or a DRY pass. The plugin syncs each one into a Claude Code skill (`/scribe:sync`), which fetches the live procedure when it is invoked. -`/graph` renders notes, tasks, and tags as a D3 force-directed graph. Tag nodes -cluster notes that share tags; invisible project hubs attract project members. Physics -controls (repulsion, link distance/strength, hub pull, gravity); click a node to peek, -click a tag to filter. +## Search and context injection + +- **Semantic search** over notes, tasks, rules, milestones and systems, with embeddings computed in-process (no external model service). It falls back to keyword search if the embedder is unavailable. +- **Context injection**: the plugin's hooks show Claude a short menu of possibly relevant records at the moments they matter: rules and prior art when a prompt arrives, before a command or a file write, and when a moment of work is reached. Claude opens a full record only when it judges it relevant. +- **Tuning**: each surface has its own thresholds and budget, in Settings → General. Every change is logged with its reason, and retrieval telemetry shows what was surfaced and what was opened. ## Claude via MCP and the plugin -The whole store is reachable by Claude through a built-in **MCP endpoint at `/mcp`** -(Bearer-auth with an API key). The **Scribe Claude Code plugin** (shipped in this -repo) wires it up: +The whole store is reachable through the MCP endpoint at `/mcp`, authenticated with an API key. The Claude Code plugin in this repo adds: -- a `SessionStart` hook that injects active-project - context so Scribe surfaces without being asked (fail-open if Scribe is unreachable); -- universal process-skills — writing-plans, systematic-debugging, verification, - brainstorming — that route their output into Scribe; -- your saved Processes auto-surfaced as skills. +- hooks for session start, each prompt, before and after tool calls, the end of a turn, compaction and session end, all failing open if Scribe is unreachable; +- skills for the working practices: using Scribe, writing plans, systematic debugging, verification, brainstorming, reporting back, reusing code, shape accounting and family canon; +- your stored processes as skills. See [API Keys & MCP](api-keys-and-mcp.md). -## Sharing and Collaboration +## The web UI -- **Share** — Share any project, note, or task with users or groups at - `viewer`/`editor`/`admin` levels from the viewer/project toolbar. -- **Groups** — Admins create platform-wide groups and assign `member`/`owner` roles; - share a resource with a group in one action. -- **Shared with me** — `/shared` lists incoming shares with permission badges. -- **Notifications** — An in-app bell (unread count, polled) fires when a project or - note is shared with you or you're added to a group. +- **Dashboard**: what to work on next. **Knowledge** browses everything. +- **Graph** (`/graph`): notes, tasks and tags as a force-directed graph, with physics controls and filtering by tag. +- **Trash**: deleted records wait here until retention purges them, and can be restored. +- **Installable** as a desktop or mobile app. -Every read and mutation is scoped by owner + direct shares + group shares. +## Sharing -## Data Export and Backup +Share a project, note or task with users or groups, as viewer, editor or admin. Admins manage groups. `/shared` lists what has been shared with you, and an in-app bell notifies you of new shares. Every read and write is checked against ownership and shares. -- **Personal export** — download all your notes/tasks as a Markdown ZIP (YAML - frontmatter) or a JSON array. -- **Admin backup** — full application backup/restore with ID remapping on - restore for cross-instance migration. -- **Scheduled backups** — a daily, dated, uncompressed `.tar` to a folder or an - S3-compatible bucket, laid out so a deduplicating tool such as restic stores - only what changed; `python -m scribe.restore` brings a fresh instance back - from one. See [Backups](backups.md). +## Export and backup -## Progressive Web App +- **Personal export**: your notes and tasks as a Markdown ZIP (with YAML frontmatter) or JSON. +- **Admin backup and restore**, with ids remapped so a backup restores onto another instance. +- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. `python -m scribe.restore` brings a fresh instance back from one. See [Backups](backups.md). -Installable as a desktop or mobile app; a service worker caches the shell. +## Sign-in -## Authentication - -Native email + password, plus optional **OIDC** sign-in (Authentik, Keycloak, etc.) -that links to a matching local account. Invite links, a registration toggle, password -reset, and session invalidation are included. See [SSO / OAuth](sso-oauth.md). +Email and password, plus optional OIDC single sign-on (Authentik, Keycloak, ...) that links to a matching local account. Invite links, password reset and signing out other sessions are included. See [SSO / OAuth](sso-oauth.md). ## Settings | Tab | Contents | |-----|----------| -| General | Instance preferences (key/value) | -| Account | Email change, password change, session invalidation | -| Profile | Per-user profile fields | -| Integrations | SearXNG status | -| Data | Personal export; backup / restore and scheduled backups (admin) | -| API Keys | Create/revoke `fmcp_` keys for the MCP endpoint | -| Config (admin) | Base URL, SMTP, DB-maintenance schedule | -| Users (admin) | User list, invite links, registration toggle | -| Logs (admin) | Error, audit, and usage logs with search | -| Groups (admin) | Create/manage groups and membership | +| General | Timezone, trash retention, retrieval thresholds, moments, reply shapes | +| Account | Email, password, active sessions | +| Profile | About you, response preferences, interests, work schedule | +| Notifications | Email notifications | +| Integrations | SearXNG, git forge connections | +| Data | Export, backup and restore, scheduled backups (admin) | +| MCP Access | API keys and the command to connect Claude | +| Config (admin) | What's running, application URL, plugin marketplace, database maintenance, SMTP, forge webhook | +| Shared areas (admin) | The canonical areas systems map to | +| Platforms (admin) | The platform catalogue family canon is grouped by | +| Users (admin) | Registration, invites, users | +| Logs (admin) | Error, audit and usage logs | +| Groups (admin) | Groups and membership | -## Keyboard Shortcuts +## Keyboard shortcuts | Key | Action | |-----|--------| -| `g` + `h` | Home (dashboard) | -| `g` + `n` | Notes | -| `g` + `t` | Knowledge (tasks) | -| `g` + `p` | Projects | -| `g` + `r` | Rulebooks | -| `g` (bare) | Graph | -| `g` + `x` | Trash | +| `g` `h` | Dashboard | +| `g` `n` | Notes | +| `g` `p` | Projects | +| `g` `r` | Rules | +| `g` `g` | Graph | +| `g` `x` | Trash | | `n` | New note | | `t` | New task | +| `e` | Edit the note you're viewing | | `/` | Focus search | -| `?` | Show shortcuts panel | -| `j` / `k` | Navigate list items | -| `Enter` | Open selected item | -| `e` | Edit current item | -| `Esc` | Close panel / blur / go home (progressive) | -| `Ctrl+S` | Save in editor | +| `?` | Show shortcuts | +| `j` / `k` | Move through a list | +| `Enter` | Open the selected item | +| `Esc` | Close the overlay, leave the field, then go to the dashboard | +| `Ctrl+S` | Save in the editor | diff --git a/docs/sso-oauth.md b/docs/sso-oauth.md index 7e3ffccb..ee336a9e 100644 --- a/docs/sso-oauth.md +++ b/docs/sso-oauth.md @@ -1,6 +1,6 @@ # OAuth / OIDC SSO Setup -Fabled Assistant supports single sign-on via any OpenID Connect provider. +Scribe supports single sign-on via any OpenID Connect provider. This guide covers Authentik, but the same pattern works with Keycloak, Authelia, Zitadel, etc. ## 1. Create the provider in Authentik @@ -8,7 +8,7 @@ This guide covers Authentik, but the same pattern works with Keycloak, Authelia, 1. Log in to the Authentik admin UI. 2. Go to **Applications → Providers → Create → OAuth2/OpenID Provider**. 3. Fill in: - - **Name:** `Fabled Assistant` (or whatever you like) + - **Name:** `Scribe` (or whatever you like) - **Authorization flow:** your default authorization flow - **Client type:** `Confidential` - **Redirect URIs:** `https://your-fabled-domain/api/auth/oauth/callback` @@ -20,7 +20,7 @@ This guide covers Authentik, but the same pattern works with Keycloak, Authelia, https://auth.example.com/application/o/fabled-assistant/ ``` -## 2. Configure Fabled Assistant +## 2. Configure Scribe Add the following environment variables to the `app` service in `docker-compose.yml`: @@ -64,7 +64,7 @@ docker compose up --build -d ## 4. Account Linking -When a user logs in via OAuth for the first time, Fabled checks in this order: +When a user logs in via OAuth for the first time, Scribe checks in this order: 1. **Existing OAuth sub** — returns that user immediately. 2. **Matching email** — if a local account already exists with the same email, the OAuth identity is linked to it automatically. The user retains all their notes and tasks. @@ -92,4 +92,4 @@ The backend will reject any `POST /api/auth/login` or `POST /api/auth/register` | Zitadel | `https://your-instance.zitadel.cloud` | | Google | `https://accounts.google.com` | -The OIDC discovery endpoint (`/.well-known/openid-configuration`) must be publicly reachable from the Fabled container (server-to-server call at login time). +The OIDC discovery endpoint (`/.well-known/openid-configuration`) must be publicly reachable from the Scribe container (server-to-server call at login time).