diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index 6e6551a..557f119 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -26,6 +26,7 @@ on: - "alembic.ini" - "Dockerfile" - "assets/**" + - "fable-mcp/**" - ".forgejo/workflows/ci.yml" # pull_request trigger intentionally omitted — all changes go through dev # first, where CI already runs on push. PR runs would be redundant duplication. diff --git a/.gitignore b/.gitignore index 507f415..c891752 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,4 @@ docker-compose.override.yml *.log .DS_Store .superpowers/ +.mcp.json diff --git a/Dockerfile b/Dockerfile index 699e156..d2f5e0e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,6 +14,13 @@ COPY pyproject.toml . COPY src/ src/ RUN pip install --no-cache-dir . +# Build the fable-mcp wheel so it can be served for download +COPY fable-mcp/ fable-mcp/ +RUN pip install --no-cache-dir build hatchling \ + && python -m build --wheel ./fable-mcp --outdir /app/dist/ \ + && pip uninstall -y build \ + && rm -rf fable-mcp/ /root/.cache/pip + COPY --from=build-frontend /build/dist/ src/fabledassistant/static/ COPY alembic.ini . COPY alembic/ alembic/ diff --git a/README.md b/README.md index 54a2346..a329927 100644 --- a/README.md +++ b/README.md @@ -2,210 +2,41 @@ A self-hosted second brain and project management application with integrated LLM capabilities. Write, organise, and act on your notes and tasks with the help of a local AI assistant — all running on your own hardware. -## What It Does +## Features -**Notes with inline formatting** — Write in Markdown with a live-preview editor (Tiptap/ProseMirror). Headings, bold, italic, lists, code blocks, and task checklists render inline. A slash-command menu (`/`) inserts common blocks without leaving the keyboard. +Notes and tasks with a Markdown editor, sub-tasks, milestones, and kanban project workspaces. AI chat with streaming responses, RAG over your notes, and tool use (web search, calendar, weather). A daily briefing that digests your tasks, RSS feeds, and weather on a schedule. Knowledge graph, per-user/group sharing, PWA with push notifications, an MCP server for external AI clients, and an Android companion app. -**Task tracking** — Notes convert freely to tasks (and back). Tasks carry status (`todo` → `in_progress` → `done`), priority, due date, sub-tasks, milestone assignment, and work logs with time tracking. +## Quick Start -**Projects and milestones** — Group related notes and tasks into projects. Milestones give projects a timeline and show completion progress. A kanban-style project view groups tasks by milestone. +**Prerequisites:** Docker and Docker Compose. 8 GB+ RAM recommended for LLM inference. -**Project Workspace** — `/workspace/:projectId` opens a three-panel environment (tasks / chat / notes) locked to a project. The AI assistant creates and updates content directly in the workspace; new notes auto-load in the editor and the task list refreshes automatically. - -**Wikilinks and backlinks** — Link notes with `[[Title]]` syntax. Click a wikilink to navigate to (or create) the referenced note. Each note shows what links to it. The editor suggests existing note titles as candidate links. - -**Tag organisation** — Tags are first-class columns (stored as a PostgreSQL array, not extracted from body text). Tag autocomplete, tag-based filtering, and a force-directed graph view show how notes cluster. - -**Knowledge graph** — `/graph` renders all notes, tasks, and tags as a D3 force-directed graph. Tag nodes cluster notes that share tags; project hub nodes (invisible) attract project members. Click any node to open a slide-in peek panel. - -**AI chat** — Full conversation history with SSE streaming. The assistant automatically retrieves semantically relevant notes (RAG) and injects them as context. Attach specific notes by paperclip for focused discussions. Useful replies can be saved directly as notes. - -**AI writing assistant** — Select a passage in the editor, give an instruction ("make this more concise", "add examples"), and stream a diff-style suggestion you can accept or reject. - -**Web research** — The assistant can search the web (SearXNG), fetch pages, and synthesise findings. Research results are saved as notes. A lightweight `search_web` tool answers quick questions inline. - -**Collaboration and sharing** — Share any project or note with other users or groups. Three permission levels: `viewer`, `editor`, `admin`. A "Shared with me" page lists all incoming shares. In-app notifications (with push) alert recipients when items are shared or when they are added to a group. - -**Groups** — Admins can create platform-wide groups, assign users roles (`member` / `owner`), and share resources with the group in one action. - -**In-app notifications** — A bell icon in the nav shows unread notification count with a 60-second polling interval. Clicking a notification navigates to the relevant resource and marks it read. - -**CalDAV calendar** — Connect an external CalDAV server (Nextcloud, Radicale, etc.) and have the assistant create, list, search, update, and delete calendar events via natural language. - -**Push notifications** — Web Push (VAPID) notifies you when AI generation completes, even in another tab. Configurable per-user from the Notifications settings tab. - -**PWA** — Installable as a desktop or mobile app. Service worker caches the shell; push is handled by `public/sw.js`. - -**Data export and backup** — Export your data as a Markdown ZIP (with YAML frontmatter) or a JSON array from the Data settings tab. Admins can export/restore full application backups (version 2 includes projects, milestones, task logs, AI drafts, note versions, push subscriptions) with proper ID remapping on restore. - -**OAuth / OIDC login** — Supports PKCE-based OIDC in addition to local username/password auth. `LOCAL_AUTH_ENABLED` can disable local login entirely for SSO-only deployments. - -**Multi-user with isolation** — All data is scoped to the owning user. Access to shared resources is resolved through `services/access.py` using the permission rank system. The first registered user becomes admin. - -**Daily Briefing** — A scheduled, dialogue-based morning briefing accessible at `/briefing`. The assistant compiles your tasks, calendar events, projects, weather forecast (Open-Meteo), and RSS feed digest at 4am, then checks in at 8am, 12pm, and 4pm. You can reply interactively — the briefing is a real conversation, not a widget. The assistant learns your preferences over time via a profile note it maintains. Configure locations, work schedule, RSS feeds, and active slots from Settings → Briefing. - -**Dark and light themes** — Defaults to dark (slate-indigo palette). One-click toggle in the header. - -**Keyboard shortcuts** — `g`+`h/n/t/p/c/g` navigate sections; `n` new note; `t` new task; `?` shows the shortcut panel. Full list available in-app. - ---- - -## Getting Started - -### Prerequisites - -- [Docker](https://docs.docker.com/get-docker/) and Docker Compose -- A machine with enough RAM to run an LLM (8 GB+ recommended for smaller models like `llama3.2`) - -### Quick Start - -1. Clone the repository: - ```bash - git clone https://github.com/your-username/fabledassistant.git - cd fabledassistant - ``` - -2. Start the application: - ```bash - docker compose up --build - ``` - -3. Open `http://localhost:5000` in your browser. - -4. Register the first user account — this account becomes the admin. - -5. Go to **Settings → General** to pull an LLM model (`llama3.2` at 2 GB is a good starting point). - -### Day-to-Day Usage - -- **Create a note** from the Notes page. Use Markdown — formatting renders live. -- **Link notes** by typing `[[` for a wikilink autocomplete dropdown. -- **Tag your notes** — add tags in the sidebar tag input; autocomplete suggests existing tags. -- **Use the AI writing assistant** — select text in the editor, write an instruction, stream a suggestion. -- **Chat with the AI** — the assistant finds relevant notes automatically. Attach specific notes for focused context. -- **Convert notes ↔ tasks** from the viewer toolbar. -- **Open a project workspace** from the project page — chat, tasks, and note editor in one view. -- **Share a project or note** — click Share in the project/note/task viewer toolbar, search for users or pick a group. -- **Manage groups** (admins) — Settings → Groups tab. -- **Backup your data** — Settings → Data tab for personal export; admin section for full application backup. - ---- - -## Configuration - -Configuration is via environment variables. See `docker-compose.yml` for defaults. - -| Variable | Default | Description | -|----------|---------|-------------| -| `DATABASE_URL` | `postgresql+asyncpg://...` | PostgreSQL connection string | -| `SECRET_KEY` | (required) | Session signing key | -| `OLLAMA_BASE_URL` | `http://ollama:11434` | Ollama API endpoint | -| `DEFAULT_MODEL` | `llama3.1` | LLM model to warm on startup | -| `EMBEDDING_MODEL` | `nomic-embed-text` | Model used for semantic search / RAG | -| `SECURE_COOKIES` | `false` | Set `true` behind TLS | -| `LOG_LEVEL` | `INFO` | Logging verbosity | -| `OIDC_ISSUER` | — | OIDC issuer URL for SSO login | -| `OIDC_CLIENT_ID` | — | OIDC client ID | -| `OIDC_CLIENT_SECRET` | — | OIDC client secret | -| `LOCAL_AUTH_ENABLED` | `true` | Set `false` to disable local login | -| `SEARXNG_URL` | — | SearXNG base URL for web search | -| `BASE_URL` | — | Public URL (used in email links and OIDC redirect) | - -For production, `docker-compose.prod.yml` supports Docker secrets (`SECRET_KEY_FILE`, `DATABASE_URL_FILE`) and includes network isolation, health checks, and resource limits. - ---- - -## Production Deployment - -### Reverse Proxy (Required) - -Fabled Assistant does **not** handle SSL/TLS. Run it behind a reverse proxy: - -- **Nginx**, **Traefik**, or **Caddy** in front of the app container -- Terminate TLS at the proxy; forward to port 5000 -- **Do not expose port 5000 directly to the internet** -- **Rate-limit auth endpoints** — recommended: ≤5 req/min per IP on `/api/auth/login` and `/api/auth/register` -- **Set CSP headers** — recommended: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'` - -### Security Checklist - -- **Strong `SECRET_KEY`** — generate with `python -c "import secrets; print(secrets.token_hex(32))"` or use Docker secrets via `SECRET_KEY_FILE` -- **Registration** — auto-closes after the first user (admin). Re-enable from Settings → Users or send invite links. -- **Session invalidation** — changing or resetting a password bumps `session_version`, evicting all other active sessions. -- **Keep Ollama on an internal network** — both compose files keep it off the host network. - ---- - -## Technical Overview - -### Stack - -| Layer | Technology | -|-------|------------| -| Frontend | Vue 3, TypeScript, Vite, Pinia, Vue Router | -| Editor | Tiptap (ProseMirror) with custom slash-command extension | -| Backend | Python 3.12, Quart (async ASGI) | -| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | -| LLM | Ollama (local) — any OpenAI-compatible API | -| Search | SearXNG (optional, self-hosted) | -| Push | Web Push / VAPID (pywebpush 2.x) | -| Deployment | Docker Compose | - -### Architecture - -The app runs as a single container serving the Vue SPA and REST API (`/api/`). The frontend is built by Vite during the Docker image build and served as static files by Quart. - -LLM interactions stream via Server-Sent Events (SSE). Chat generation runs in background `asyncio` tasks with an in-memory event buffer supporting client reconnection without data loss. An abort mechanism lets users cancel in-flight generations. - -Semantic search (RAG) uses `nomic-embed-text` via Ollama to generate embeddings stored in PostgreSQL. Notes above 0.60 cosine similarity are auto-injected into the system prompt; notes between 0.45–0.60 appear as sidebar suggestions. - -Permission resolution is centralised in `services/access.py`. All resource access goes through `get_project_permission` / `get_note_permission`, which check ownership, direct shares, group membership shares, and note→project inheritance in order, returning the highest applicable permission. - -### Database - -PostgreSQL with SQLAlchemy 2.0 async. Tasks are notes with non-null `status` (unified `Note` model). Tags are stored as a `ARRAY[text]` column. Migrations run automatically on startup via Alembic. - -### Project Structure - -``` -fabledassistant/ -├── docker-compose.yml # Development stack -├── docker-compose.prod.yml # Production stack (Docker Swarm) -├── Dockerfile # Multi-stage build (Node → Python) -├── alembic/ # Database migrations -├── src/fabledassistant/ -│ ├── app.py # Quart app factory + blueprint registration -│ ├── models/ # SQLAlchemy models -│ ├── routes/ # API blueprints -│ ├── services/ # Business logic (access, sharing, groups, …) -│ └── static/ # Built frontend (generated at build time) -└── frontend/ - └── src/ - ├── views/ # Page-level components - ├── components/ # Reusable UI components - ├── composables/ # Vue composables (autosave, shortcuts, …) - ├── stores/ # Pinia stores (auth, chat, notes, notifications, …) - └── api/ # Typed API client (client.ts) -``` - -### Development - -All development is done via Docker. No local dependency installation required. +Download [`docker-compose.quickstart.yml`](docker-compose.quickstart.yml) from this repo, then: ```bash -# Start the dev stack (hot-reload not included — rebuild on changes) -docker compose up --build +# Optional but recommended — set a secret key +export SECRET_KEY=your-random-secret-here -# Reset the database -docker compose down -v && docker compose up --build - -# Lint, format, typecheck, test (runs inside Docker via Makefile) -make check +docker compose -f docker-compose.quickstart.yml up -d ``` -CI runs on Forgejo Actions with a custom runner image (`py3.12-node22`) that has Python 3.12 and Node 22 pre-installed. Pushes to `dev` build a `:dev` image; merges to `main` build `:latest`. +Open `http://localhost:5000`. The first user to register becomes admin. Go to **Settings → General** to pull an LLM model — `qwen3:8b` or `llama3.1:8b` are good starting points. ---- +> **GPU:** Ollama runs CPU-only by default. See the comments in `docker-compose.quickstart.yml` to enable NVIDIA GPU passthrough. + +> **Development:** To build from source, see [Development](docs/development.md). + +## Documentation + +| Doc | Contents | +|-----|----------| +| [Architecture](docs/architecture.md) | Stack, design decisions, data models, key services | +| [Configuration](docs/configuration.md) | Environment variables, Docker Compose, production setup, security | +| [Features](docs/features.md) | Detailed feature breakdown and keyboard shortcuts | +| [Development](docs/development.md) | Dev workflow, CI/CD, migrations, release process | +| [API Keys & MCP](docs/api-keys-and-mcp.md) | API key management and Fable MCP install guide | +| [SSO / OAuth](docs/sso-oauth.md) | OIDC setup for Authentik, Keycloak, and other providers | +| [API Reference](docs/api-reference.md) | All REST API endpoints | +| [Android App](docs/android-app.md) | Flutter companion app architecture and feature status | ## License diff --git a/alembic/versions/0028_add_briefing_improvements.py b/alembic/versions/0028_add_briefing_improvements.py new file mode 100644 index 0000000..d7d35b8 --- /dev/null +++ b/alembic/versions/0028_add_briefing_improvements.py @@ -0,0 +1,55 @@ +"""Add briefing improvements: rss_items topics/classified_at, messages metadata, + rss_item_reactions, briefing_task_snapshot.""" + +from alembic import op + +revision = "0028" +down_revision = "0027" + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE rss_items + ADD COLUMN IF NOT EXISTS topics TEXT[] DEFAULT '{}', + ADD COLUMN IF NOT EXISTS classified_at TIMESTAMPTZ + """) + op.execute(""" + ALTER TABLE messages + ADD COLUMN IF NOT EXISTS metadata JSONB + """) + op.execute(""" + CREATE TABLE IF NOT EXISTS rss_item_reactions ( + id SERIAL PRIMARY KEY, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + rss_item_id INTEGER NOT NULL REFERENCES rss_items(id) ON DELETE CASCADE, + reaction TEXT NOT NULL CHECK (reaction IN ('up', 'down')), + created_at TIMESTAMPTZ DEFAULT NOW(), + UNIQUE (user_id, rss_item_id) + ) + """) + op.execute( + "CREATE INDEX IF NOT EXISTS ix_rss_item_reactions_user_id " + "ON rss_item_reactions(user_id)" + ) + op.execute(""" + CREATE TABLE IF NOT EXISTS briefing_task_snapshot ( + id SERIAL PRIMARY KEY, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + task_id INTEGER NOT NULL REFERENCES notes(id) ON DELETE CASCADE, + snapshot_hash TEXT NOT NULL, + last_briefed TIMESTAMPTZ DEFAULT NOW(), + UNIQUE (user_id, task_id) + ) + """) + op.execute( + "CREATE INDEX IF NOT EXISTS ix_briefing_task_snapshot_user_id " + "ON briefing_task_snapshot(user_id)" + ) + + +def downgrade() -> None: + op.execute("DROP TABLE IF EXISTS briefing_task_snapshot") + op.execute("DROP TABLE IF EXISTS rss_item_reactions") + op.execute("ALTER TABLE messages DROP COLUMN IF EXISTS metadata") + op.execute("ALTER TABLE rss_items DROP COLUMN IF EXISTS classified_at") + op.execute("ALTER TABLE rss_items DROP COLUMN IF EXISTS topics") diff --git a/alembic/versions/0029_add_calendar_columns.py b/alembic/versions/0029_add_calendar_columns.py new file mode 100644 index 0000000..1e663f8 --- /dev/null +++ b/alembic/versions/0029_add_calendar_columns.py @@ -0,0 +1,19 @@ +"""Add caldav_uid and color columns to events table.""" + +from alembic import op + +revision = "0029" +down_revision = "0028" + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE events + ADD COLUMN IF NOT EXISTS caldav_uid TEXT DEFAULT '', + ADD COLUMN IF NOT EXISTS color TEXT DEFAULT '' + """) + + +def downgrade() -> None: + op.execute("ALTER TABLE events DROP COLUMN IF EXISTS color") + op.execute("ALTER TABLE events DROP COLUMN IF EXISTS caldav_uid") diff --git a/alembic/versions/0030_rag_scoping.py b/alembic/versions/0030_rag_scoping.py new file mode 100644 index 0000000..38858a3 --- /dev/null +++ b/alembic/versions/0030_rag_scoping.py @@ -0,0 +1,23 @@ +"""Add rag_project_id to conversations; auto_summary columns to projects.""" + +from alembic import op + +revision = "0030" +down_revision = "0029" + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE conversations + ADD COLUMN IF NOT EXISTS rag_project_id INTEGER DEFAULT NULL + """) + op.execute(""" + ALTER TABLE projects + ADD COLUMN IF NOT EXISTS auto_summary TEXT DEFAULT NULL, + ADD COLUMN IF NOT EXISTS summary_updated_at TIMESTAMPTZ DEFAULT NULL + """) + + +def downgrade() -> None: + op.execute("ALTER TABLE conversations DROP COLUMN IF EXISTS rag_project_id") + op.execute("ALTER TABLE projects DROP COLUMN IF EXISTS auto_summary, DROP COLUMN IF EXISTS summary_updated_at") diff --git a/docker-compose.quickstart.yml b/docker-compose.quickstart.yml new file mode 100644 index 0000000..a5a294f --- /dev/null +++ b/docker-compose.quickstart.yml @@ -0,0 +1,82 @@ +# Fabled Assistant — Quick Start +# +# No build required. Pulls the latest pre-built image from the registry. +# +# Usage: +# 1. Download this file +# 2. docker compose -f docker-compose.quickstart.yml up -d +# 3. Open http://localhost:5000 — the first account registered becomes admin +# 4. Go to Settings → General to pull an LLM model (qwen3:8b or llama3.1:8b are good starting points) +# +# Set SECRET_KEY via environment variable or a .env file alongside this file: +# SECRET_KEY=your-random-secret-here + +services: + app: + image: git.fabledsword.com/bvandeusen/fabledassistant:latest + ports: + - "5000:5000" + environment: + DATABASE_URL: "postgresql+asyncpg://fabled:fabled@db:5432/fabledassistant" + SECRET_KEY: "${SECRET_KEY:-change-me-in-production}" + OLLAMA_URL: "http://ollama:11434" + OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.1:8b}" + LOG_LEVEL: "${LOG_LEVEL:-INFO}" + volumes: + - app_data:/data + depends_on: + db: + condition: service_healthy + ollama: + condition: service_healthy + restart: unless-stopped + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:5000/api/health')"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + + db: + image: postgres:16-alpine + volumes: + - pgdata:/var/lib/postgresql/data + environment: + POSTGRES_USER: fabled + POSTGRES_PASSWORD: fabled + POSTGRES_DB: fabledassistant + healthcheck: + test: ["CMD-SHELL", "pg_isready -U fabled"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + + ollama: + image: ollama/ollama + volumes: + - ollama_models:/root/.ollama + environment: + OLLAMA_MAX_LOADED_MODELS: "2" + OLLAMA_KEEP_ALIVE: "30m" + OLLAMA_FLASH_ATTENTION: "1" + healthcheck: + test: ["CMD-SHELL", "ollama list > /dev/null 2>&1"] + interval: 30s + timeout: 10s + retries: 5 + start_period: 15s + restart: unless-stopped + # Uncomment to enable NVIDIA GPU passthrough (requires nvidia-container-toolkit): + # deploy: + # resources: + # reservations: + # devices: + # - driver: nvidia + # count: all + # capabilities: [gpu] + +volumes: + app_data: + pgdata: + ollama_models: diff --git a/docker-compose.yml b/docker-compose.yml index 1879389..5cf9bb6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -55,13 +55,14 @@ services: OLLAMA_NUM_PARALLEL: "2" OLLAMA_KEEP_ALIVE: "30m" OLLAMA_FLASH_ATTENTION: "1" - deploy: - resources: - reservations: - devices: - - driver: nvidia - count: all - capabilities: [gpu] + # GPU reservation commented out — no nvidia-container-toolkit on this host + # deploy: + # resources: + # reservations: + # devices: + # - driver: nvidia + # count: all + # capabilities: [gpu] volumes: pgdata: diff --git a/docs/2026-03-23-fable-mcp-plan.md b/docs/2026-03-23-fable-mcp-plan.md index 1e2bf69..8071a2e 100644 --- a/docs/2026-03-23-fable-mcp-plan.md +++ b/docs/2026-03-23-fable-mcp-plan.md @@ -6,6 +6,8 @@ **Architecture:** Phase 1 adds bearer token auth to the existing Quart app (new `api_keys` table, updated `_check_auth`, settings UI tab). Phase 2 is a standalone `fable-mcp/` Python package using the `mcp[cli]` SDK that calls the Fable HTTP API via `httpx`. The two phases are sequential — Phase 2 can be built/tested independently using a write-scoped API key once Phase 1 is done. +**Deployment decision:** `fable-mcp/` stays permanently inside the `fabledassistant` repo. It will always be versioned alongside the backend it targets. Future Task A will serve the package from the running Fable Docker image so users can install it directly from their instance. + **Tech Stack:** Python 3.12, Quart (Phase 1); `mcp[cli]`, `httpx`, `python-dotenv` (Phase 2); Vue 3 + TypeScript (Settings UI); pytest for both. **Testing convention:** Phase 1 tests run via `docker compose run --rm app pytest tests/ -v`. Phase 2 tests run locally with `cd fable-mcp && pytest -v`. All Phase 1 tests are unit tests of pure functions — mock DB calls with `unittest.mock.AsyncMock`; never connect to a real database in tests. diff --git a/docs/android-app.md b/docs/android-app.md new file mode 100644 index 0000000..ef9c0ee --- /dev/null +++ b/docs/android-app.md @@ -0,0 +1,73 @@ +# Android Companion App + +The Android companion app lives in a separate repository at `/home/bvandeusen/Nextcloud/Projects/fabled_app`. + +## Stack + +- Flutter + Dart +- Riverpod (state management) +- GoRouter (navigation) +- Dio (HTTP client) +- PersistCookieJar (session persistence) +- SSE streaming via `fetch` + `ReadableStream` bridge + +## Architecture + +``` +lib/ + app.dart # GoRouter + _Shell + _QuickCaptureBar + core/constants.dart # Routes.* + data/ + models/ # note.dart, task.dart, project.dart + api/ # notes_api.dart, tasks_api.dart, projects_api.dart + repositories/ # notes, tasks, projects repositories + providers/ + api_client_provider.dart # all API + repository providers + notes_provider.dart # NotesNotifier + tasks_provider.dart # TasksNotifier + projects_provider.dart # ProjectsNotifier + screens/ + notes/note_edit_screen.dart # chip tag input + ProjectSelector + tasks/task_edit_screen.dart # ProjectSelector + projects/project_list_screen.dart + widgets/ + project_selector.dart # reusable DropdownButtonFormField +``` + +## Navigation + +4-tab shell (Notes · Tasks · Projects · Chat): +- Phone: bottom `NavigationBar` +- Tablet/landscape: `NavigationRail` + +Quick Capture bar persists across all tabs. Settings accessible from top-right icon. + +## Feature Status + +| Feature | Status | Notes | +|---------|--------|-------| +| Notes CRUD | ✅ | Tags chip input; project selector in editor | +| Tasks CRUD | ✅ | Project selector in editor | +| Projects list | ✅ | Active/archived sections; long-press status change; create dialog | +| Chat + SSE | ✅ | Full streaming | +| Quick Capture | ✅ | Offline queue with retry | +| Tags | ✅ | Chip input in NoteEditScreen; typed as `List` | +| Project assignment | ✅ | `ProjectSelector` dropdown in Note + Task editors | +| Milestones | ❌ deferred | Too granular for mobile; web UI handles it | +| Push notifications | ❌ incompatible | Backend uses browser VAPID; Flutter needs FCM/APNs — separate implementation required | +| CalDAV settings | ❌ intentional | Server-side config only; not exposed in mobile app | + +## API Compatibility Notes + +- `GET /api/projects/:id` returns a flat JSON object (not `{project: ...}` wrapper); includes `summary` field. +- `POST /api/projects` returns the project dict directly (201). +- `PATCH /api/projects/:id` returns the updated project dict. +- Task body field is `body` (not `description`) — the app maps `description` → `body` on serialize. + +## Self-Update + +The app supports self-update via the Forgejo release API (`update_provider.dart`). It checks the latest release tag and prompts the user to download and install a new APK when one is available. + +## CI + +Builds are triggered from the Forgejo Actions pipeline in the `fabled_app` repository. The APK is attached to the release as a downloadable artifact. diff --git a/docs/api-keys-and-mcp.md b/docs/api-keys-and-mcp.md new file mode 100644 index 0000000..61f058e --- /dev/null +++ b/docs/api-keys-and-mcp.md @@ -0,0 +1,144 @@ +# API Keys and Fable MCP + +## API Keys + +API keys let external tools access your Fable data without a browser session. Each key is scoped to a single user — it can only access data that user owns or has been shared with them. + +### Scopes + +| Scope | Permissions | +|-------|-------------| +| `read` | GET endpoints only — list, search, fetch content | +| `write` | Full read + create, update, delete | + +Admin-level operations (log access, user management) require a `write`-scoped key from an admin account. + +### Creating a Key + +1. Go to **Settings → API Keys** +2. Enter a name (e.g. "Claude MCP", "Home Server") +3. Choose scope +4. Click **Generate Key** +5. Copy the key immediately — it is shown only once + +After creation you can download: +- **`.env` file** — `FABLE_URL` + `FABLE_API_KEY` ready to paste +- **Claude config JSON** — `mcpServers` block ready to merge into `~/.claude.json` + +### Revoking a Key + +Click **Revoke** next to the key in the API Keys table and confirm. Revoked keys are deleted immediately. + +--- + +## Fable MCP Server + +The Fable MCP server (`fable-mcp`) exposes Fable as a set of MCP tools that Claude (and other MCP clients) can use to read and write your notes, tasks, projects, and more. + +### Download + +The wheel is bundled into the Docker image at build time and available for download from **Settings → API Keys → Fable MCP** when you are logged in. + +You can also download it directly: +``` +GET /api/fable-mcp/download +``` +(Requires login — authenticated browser session or API key in `Authorization: Bearer ` header.) + +### Installation + +```bash +# Install the wheel +pip install fable_mcp-*.whl + +# Verify +fable-mcp --help +``` + +### Configuration + +The server reads two environment variables: + +| Variable | Description | +|----------|-------------| +| `FABLE_URL` | Base URL of your Fable instance (e.g. `https://notes.example.com`) | +| `FABLE_API_KEY` | API key generated from Settings → API Keys | + +Create a `.env` file in your working directory, or set them in your shell / MCP config. + +### Claude Code (Global) + +Add to `~/.claude.json`: + +```json +{ + "mcpServers": { + "fable": { + "type": "stdio", + "command": "fable-mcp", + "env": { + "FABLE_URL": "https://your-fable-instance.example.com", + "FABLE_API_KEY": "your-api-key" + } + } + } +} +``` + +### Claude Code (Project-scoped) + +Add a `.mcp.json` at the project root (same format as the global config). Project-scoped config takes precedence over global when the same server name is defined in both. This is useful for using a dev instance or admin key within a specific project. + +```json +{ + "mcpServers": { + "fable": { + "type": "stdio", + "command": "fable-mcp", + "env": { + "FABLE_URL": "http://localhost:5000", + "FABLE_API_KEY": "your-dev-api-key" + } + } + } +} +``` + +Note: `.mcp.json` contains an API key and should be added to `.gitignore`. + +### Available Tools + +| Tool | Description | +|------|-------------| +| `fable_list_notes` | List notes, filter by tag or search text | +| `fable_get_note` | Fetch a note by ID | +| `fable_create_note` | Create a new note | +| `fable_update_note` | Update a note | +| `fable_delete_note` | Delete a note | +| `fable_list_tasks` | List tasks, filter by status or project | +| `fable_get_task` | Fetch a task by ID | +| `fable_create_task` | Create a new task | +| `fable_update_task` | Update a task | +| `fable_add_task_log` | Append a work log entry to a task | +| `fable_list_projects` | List all projects | +| `fable_get_project` | Fetch a project with milestone summary | +| `fable_create_project` | Create a project | +| `fable_update_project` | Update a project | +| `fable_list_milestones` | List milestones for a project | +| `fable_create_milestone` | Create a milestone | +| `fable_update_milestone` | Update a milestone | +| `fable_search` | Semantic search over notes and tasks | +| `fable_list_conversations` | List MCP chat conversations | +| `fable_send_message` | Send a message to Fable's LLM | +| `fable_get_app_logs` | Fetch application logs (admin key required) | + +### Development Notes + +The `fable-mcp` package lives in `fable-mcp/` in this repository. The Docker build compiles it into a wheel at `/app/dist/` so it can be served for download without requiring the source tree at runtime. + +To build the wheel locally: +```bash +cd fable-mcp +pip install build hatchling +python -m build --wheel . +``` diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 0000000..1a571ca --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,244 @@ +# API Reference + +All endpoints require login (session cookie or `Authorization: Bearer `) unless marked **(public)**. + +## Health + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/health` | Health check **(public)** | + +## Auth + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/auth/status` | `{has_users, registration_open, oauth_enabled, local_auth_enabled}` **(public)** | +| POST | `/api/auth/register` | Register new user (first user becomes admin; 403 if registration closed or local auth disabled) | +| POST | `/api/auth/login` | Login with username/password (403 if local auth disabled) | +| POST | `/api/auth/logout` | Clear session | +| GET | `/api/auth/me` | Current user info (includes `has_password: bool`) | +| PUT | `/api/auth/password` | Change password `{current_password, new_password}` | +| PUT | `/api/auth/email` | Change email `{email, password?}` (password required only for local-auth users) | +| POST | `/api/auth/invalidate-sessions` | Bump `session_version` — evicts all other sessions, keeps current alive | +| POST | `/api/auth/forgot-password` | Send password reset email `{email}` | +| POST | `/api/auth/reset-password` | Reset password with token `{token, new_password}` | +| GET | `/api/auth/oauth/login` | Initiate OIDC PKCE flow → redirect to provider | +| GET | `/api/auth/oauth/callback` | OIDC callback — exchange code, find/create user, redirect to `/` | +| GET | `/api/auth/invitation/:token` | Validate invitation token **(public)** | +| POST | `/api/auth/register-with-invite` | Register with token `{token, username, password}` **(public)** | + +## Notes + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/notes` | List notes. Params: `q`, `tag`, `sort`, `order`, `limit`, `offset`, `project_id`, `milestone_id`, `parent_id`, `type` (`note`/`task`/`all`) | +| POST | `/api/notes` | Create note `{title, body, tags?, status?, priority?, due_date?, project_id?, milestone_id?, parent_id?}` | +| GET | `/api/notes/tags` | All tags (param: `q` for filter) | +| POST | `/api/notes/suggest-tags` | LLM tag suggestions `{title, body, current_tags?}` → `{suggested_tags}` | +| POST | `/api/notes/link-suggestions` | Detect note titles as plain text in body `{body, project_id, exclude_note_id}` → `[{note_id, title, count}]` | +| GET | `/api/notes/by-title` | Resolve note by exact title (param: `title`) | +| POST | `/api/notes/resolve-title` | Get-or-create note by title `{title}` (wikilink click) | +| GET | `/api/notes/:id` | Get single note | +| PUT | `/api/notes/:id` | Full update | +| PATCH | `/api/notes/:id` | Partial update (same fields as PUT) | +| DELETE | `/api/notes/:id` | Delete note | +| POST | `/api/notes/:id/convert-to-task` | Set `status='todo'`, `priority='none'` | +| POST | `/api/notes/:id/convert-to-note` | Clear `status`, `priority`, `due_date` | +| POST | `/api/notes/:id/append-tag` | Add tag `{tag}` → updated note | +| GET | `/api/notes/:id/backlinks` | Notes/tasks with `[[Title]]` references to this note | +| GET | `/api/notes/:id/versions` | List note version history | +| GET | `/api/notes/:id/versions/:vid` | Get a specific version | +| GET | `/api/notes/:id/draft` | Get current AI draft | +| PUT | `/api/notes/:id/draft` | Save AI draft | +| DELETE | `/api/notes/:id/draft` | Delete AI draft | +| POST | `/api/notes/assist` | Launch AI assist generation → 202 `{body, target_section?, instruction, whole_doc?}` | +| GET | `/api/notes/assist/stream` | SSE stream for assist (Last-Event-ID reconnect; events: `chunk`, `done`, `error`) | + +## Tasks + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/tasks` | List tasks. Params: `q`, `tag`, `status`, `priority`, `due_before`, `due_after`, `sort`, `order`, `limit`, `offset` | +| POST | `/api/tasks` | Create task (accepts `project` name string → resolved to `project_id`) | +| GET | `/api/tasks/:id` | Get task (includes `parent_title`) | +| PUT | `/api/tasks/:id` | Full update | +| PATCH | `/api/tasks/:id/status` | Quick status update `{status}` | +| DELETE | `/api/tasks/:id` | Delete task | +| GET | `/api/tasks/:id/logs` | List work logs | +| POST | `/api/tasks/:id/logs` | Create log `{content, duration_minutes?}` | +| PATCH | `/api/tasks/:id/logs/:log_id` | Update log | +| DELETE | `/api/tasks/:id/logs/:log_id` | Delete log | + +## Projects & Milestones + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/projects` | List projects (owned + shared) | +| POST | `/api/projects` | Create project | +| GET | `/api/projects/:id` | Get project with `milestone_summary` | +| PATCH | `/api/projects/:id` | Update project | +| DELETE | `/api/projects/:id` | Delete project | +| GET | `/api/projects/:id/notes` | Notes + tasks in this project | +| GET | `/api/projects/:id/milestones` | List milestones | +| POST | `/api/projects/:id/milestones` | Create milestone | +| PATCH | `/api/projects/:id/milestones/:mid` | Update milestone | +| DELETE | `/api/projects/:id/milestones/:mid` | Delete milestone | + +## Sharing + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/projects/:id/shares` | List project shares | +| POST | `/api/projects/:id/shares` | Create project share `{user_id?, group_id?, permission}` | +| PATCH | `/api/projects/:id/shares/:sid` | Update permission | +| DELETE | `/api/projects/:id/shares/:sid` | Remove share | +| GET | `/api/notes/:id/shares` | List note shares | +| POST | `/api/notes/:id/shares` | Create note share | +| PATCH | `/api/notes/:id/shares/:sid` | Update permission | +| DELETE | `/api/notes/:id/shares/:sid` | Remove share | +| GET | `/api/shared-with-me` | All resources shared with the current user | + +## Groups + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/groups` | List all groups (admin only) | +| POST | `/api/groups` | Create group `{name, description?}` | +| PATCH | `/api/groups/:id` | Update group | +| DELETE | `/api/groups/:id` | Delete group | +| GET | `/api/groups/:id/members` | List members | +| POST | `/api/groups/:id/members` | Add member `{user_id, role}` | +| PATCH | `/api/groups/:id/members/:uid` | Update member role | +| DELETE | `/api/groups/:id/members/:uid` | Remove member | + +## Chat + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/chat/conversations` | List conversations (params: `limit`, `offset`) | +| POST | `/api/chat/conversations` | Create conversation `{title?, model?}` | +| POST | `/api/chat/conversations/bulk-delete` | Delete multiple conversations `{ids: number[]}` | +| GET | `/api/chat/conversations/:id` | Get conversation with all messages | +| PATCH | `/api/chat/conversations/:id` | Update title or model | +| DELETE | `/api/chat/conversations/:id` | Delete conversation (cascades to messages) | +| POST | `/api/chat/conversations/:id/messages` | Start generation → 202. Body: `{content, context_note_id?, include_note_ids?, rag_project_id?, workspace_project_id?, think?}` | +| GET | `/api/chat/conversations/:id/generation/stream` | SSE stream (Last-Event-ID reconnect; events: `context`, `chunk`, `tool_call`, `status`, `done`, `error`) | +| POST | `/api/chat/conversations/:id/generation/cancel` | Cancel active generation | +| POST | `/api/chat/messages/:id/save-as-note` | Save assistant message as note | +| POST | `/api/chat/conversations/:id/summarize` | Summarize conversation → note | +| GET | `/api/chat/status` | Ollama availability + model state `{ollama, model, default_model}` | +| GET | `/api/chat/models` | List installed Ollama models (includes `loaded: bool`, `modified_at`) | +| POST | `/api/chat/models/pull` | Pull model (SSE NDJSON progress) `{model}` | +| POST | `/api/chat/models/delete` | Delete model `{model}` | +| GET | `/api/chat/ps` | Currently loaded (hot) models | +| POST | `/api/chat/warm` | Pre-load model into VRAM `{model}` → 202 | + +## Quick Capture + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/quick-capture` | Classify + create item from natural language `{text}` → `{success, type, message, data}` | + +## Search + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/search` | Semantic + keyword search across notes and tasks. Params: `q`, `type` (`note`/`task`/`all`), `limit` | + +## Briefing + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/briefing/config` | Get briefing configuration | +| PUT | `/api/briefing/config` | Save briefing configuration | +| GET | `/api/briefing/feeds` | List RSS feeds | +| POST | `/api/briefing/feeds` | Add RSS feed `{url, name?, category?}` | +| DELETE | `/api/briefing/feeds/:id` | Delete feed | +| POST | `/api/briefing/feeds/refresh` | Trigger immediate feed refresh → `{feeds_refreshed, new_items}` | +| GET | `/api/briefing/weather` | Get weather configuration | +| PUT | `/api/briefing/weather` | Save weather locations | +| POST | `/api/briefing/weather/geocode` | Geocode address `{query}` → `{lat, lon, label}` | +| POST | `/api/briefing/trigger` | Manually fire a briefing slot `{slot}` | +| GET | `/api/briefing/conversations` | List past briefing conversations | +| GET | `/api/briefing/conversations/today` | Get/create today's briefing conversation | +| GET | `/api/briefing/conversations/:id/messages` | Get messages for a briefing conversation | + +## Settings + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/settings` | All settings as `{key: value}` | +| PUT | `/api/settings` | Update settings `{key: value, ...}` | +| GET | `/api/settings/models` | Installed models + defaults | +| GET | `/api/settings/search` | Proxy SearXNG search (params: `q`) | + +## API Keys + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/api-keys` | List user's API keys | +| POST | `/api/api-keys` | Create key `{name, scope}` → `{key, ...}` (key shown once) | +| DELETE | `/api/api-keys/:id` | Revoke key | + +## Fable MCP Distribution + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/fable-mcp/info` | `{available: bool, filename: string\|null}` | +| GET | `/api/fable-mcp/download` | Download wheel file | + +## Notifications + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/notifications` | List notifications | +| GET | `/api/notifications/count` | Unread count | +| POST | `/api/notifications/:id/read` | Mark read | +| POST | `/api/notifications/read-all` | Mark all read | + +## Push + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/push/vapid-public-key` | VAPID public key for subscription | +| POST | `/api/push/subscribe` | Register push subscription | +| DELETE | `/api/push/subscribe` | Unregister push subscription | + +## Images + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/images/:id` | Serve cached image **(no auth required — IDs are opaque SHA-256)** | + +## Users + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/users/search` | Search users by username/email prefix (param: `q`, min 2 chars, excludes self) | + +## Export + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/export` | Export data. Params: `format=markdown` (ZIP with `.md` + YAML frontmatter) or `format=json` | + +## Admin + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/admin/backup` | Export backup (`?scope=user` for own data; full requires admin) | +| POST | `/api/admin/restore` | Restore from JSON backup | +| GET | `/api/admin/users` | List all users | +| DELETE | `/api/admin/users/:id` | Delete user (cannot delete self) | +| GET | `/api/admin/registration` | Get registration open/closed state | +| PUT | `/api/admin/registration` | Toggle registration `{open: bool}` | +| POST | `/api/admin/invitations` | Create invitation `{email}` → sends email | +| GET | `/api/admin/invitations` | List pending invitations | +| DELETE | `/api/admin/invitations/:id` | Revoke invitation | +| GET | `/api/admin/logs` | Log entries. Params: `category`, `user_id`, `search`, `date_from`, `date_to`, `limit`, `offset` | +| GET | `/api/admin/logs/stats` | Log category counts | +| GET | `/api/admin/base-url` | Get base URL setting | +| PUT | `/api/admin/base-url` | Set base URL `{base_url}` | +| GET | `/api/admin/smtp` | Get SMTP config (password masked) | +| PUT | `/api/admin/smtp` | Save SMTP config | +| POST | `/api/admin/smtp/test` | Send test email `{recipient}` | diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..91ab316 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,384 @@ +# Architecture + +## Stack + +| Layer | Technology | Notes | +|-------|-----------|-------| +| Frontend | Vue 3 + TypeScript + Vite + Pinia + Vue Router | SPA served from the same container as the API | +| Editor | Tiptap (ProseMirror) with custom slash-command extension | | +| Backend | Python 3.12, Quart (async ASGI) | Serves both API and built frontend static files | +| Database | PostgreSQL 16, SQLAlchemy 2.0 async, Alembic | asyncpg driver | +| LLM | Ollama (local) | Any OpenAI-compatible API also works | +| Search | SearXNG (optional, self-hosted) | Web search + image search | +| Push | Web Push / VAPID (pywebpush 2.x) | | +| Deployment | Docker Compose | Single-container app + separate DB + LLM service | + +## High-Level Component Diagram + +``` +┌─────────────────────────────────────────────┐ +│ Docker Compose │ +│ │ +│ ┌──────────────────────┐ ┌────────────┐ │ +│ │ fabledassistant │ │ ollama │ │ +│ │ ┌────────────────┐ │ │ │ │ +│ │ │ Quart Server │ │ │ LLM API │ │ +│ │ │ ┌──────────┐ │ │ │ │ │ +│ │ │ │ Vue SPA │ │ │ └────────────┘ │ +│ │ │ │ (static) │ │ │ ▲ │ +│ │ │ └──────────┘ │ │ │ │ +│ │ │ ┌──────────┐ │ │ HTTP/REST │ +│ │ │ │ /api/* │──┼──┼─────────┘ │ +│ │ │ └──────────┘ │ │ │ +│ │ │ │ │ │ ┌────────────┐ │ +│ │ │ ▼ │ │ │ PostgreSQL │ │ +│ │ │ ┌──────────┐ │ │ │ 16 │ │ +│ │ │ │ asyncpg │──┼──┼──▶ │ │ +│ │ │ └──────────┘ │ │ └────────────┘ │ +│ │ └────────────────┘ │ │ +│ └──────────────────────┘ │ +└─────────────────────────────────────────────┘ +``` + +## Project Structure + +``` +fabledassistant/ +├── docker-compose.yml # Development stack +├── docker-compose.prod.yml # Production stack (Docker Swarm) +├── Dockerfile # Multi-stage build (Node → Python) +├── alembic/ # Database migrations +│ └── versions/ # Migration files (idempotent raw SQL) +├── fable-mcp/ # Fable MCP server package +│ └── fable_mcp/ +│ ├── server.py # FastMCP tool registrations +│ ├── client.py # FableClient (httpx wrapper) +│ └── tools/ # Tool modules (notes, tasks, projects, …) +├── src/fabledassistant/ +│ ├── app.py # Quart app factory + blueprint registration +│ ├── config.py # Config class (reads env vars) +│ ├── auth.py # login_required decorator, session checks +│ ├── models/ # SQLAlchemy models +│ ├── routes/ # API blueprints (one file per resource) +│ ├── services/ # Business logic (access, llm, tools, sharing, …) +│ └── static/ # Built Vue SPA (generated at Docker build time) +└── frontend/ + └── src/ + ├── views/ # Page-level Vue components + ├── components/ # Reusable UI components + ├── composables/ # Vue composables (autosave, shortcuts, …) + ├── stores/ # Pinia stores (auth, chat, notes, notifications, …) + └── api/ # Typed API client (client.ts) +``` + +## Key Design Decisions + +**Single container for frontend + API.** Quart serves the Vue.js production build as static files and exposes the REST API under `/api/`. The SPA is built by Vite during the Docker image build. + +**SPA routing via 404 handler.** `app.py` uses `@app.errorhandler(404)` (not a catch-all route) to serve static files or fall back to `index.html`. API routes (`/api/*`) always return JSON 404. This avoids a catch-all `/` route intercepting API GETs. + +**Unified note/task model.** A task is just a note with task attributes enabled. `status IS NOT NULL` means it's a task. "Convert to task" sets `status='todo'`; "convert to note" clears `status`, `priority`, `due_date`. No separate table, no cascade complexity. + +**First-class tag column.** Tags live in a `tags ARRAY[text]` column and are explicitly set by the client — not auto-extracted from body text. Hierarchical tags (`project/webapp`) supported via SQL `unnest + LIKE` prefix matching. + +**Background generation architecture.** LLM streaming runs in a detached `asyncio.Task` that writes into an in-memory `GenerationBuffer`. SSE clients tail the buffer and can reconnect mid-stream without data loss. Buffer has a `cancel_event` for user-initiated stop. Completed buffers are cleaned up after 60s grace period. Periodic DB flushes every 5s preserve partial content. Both chat and AI Assist use this architecture. + +**SSE over WebSockets for LLM streaming.** SSE clients connect via `GET /api/chat/conversations/:id/generation/stream` with `Last-Event-ID` reconnection support. Frontend uses `fetch()` + `ReadableStream`. + +**Context building is server-side.** Backend fetches URL content and searches notes. Frontend sends the message text + optional context note IDs. `build_context()` returns `(messages, context_meta)`; metadata includes auto-found note IDs/titles sent to frontend via a `context` SSE event before streaming begins. + +**No blocking long-running operations.** Any slow operation (model pulls, LLM calls, URL fetching) must never block app startup or freeze the UI. Backend uses SSE streaming for incremental responses. Model pulls stream NDJSON progress to the frontend. + +**SSRF protection.** `services/llm.py` blocks requests to loopback, private, link-local, reserved, and multicast addresses before fetching. `follow_redirects=False` prevents redirect-based bypasses. + +**Session cookie security.** `HttpOnly` and `SameSite=Lax` always set. `Secure` flag controlled by `SECURE_COOKIES` env var. + +**Rate limiting.** In-memory sliding-window rate limiter (`rate_limit.py`) applied to auth endpoints: login (10/60s), register (5/300s), forgot-password (5/300s), reset-password (10/60s). Keys are per-IP. `TRUST_PROXY_HEADERS` env var enables `X-Forwarded-For` / `X-Real-IP` when behind a reverse proxy. + +**Idempotent migrations.** All Alembic migrations use raw SQL with `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DO $$ BEGIN CREATE TYPE ... EXCEPTION WHEN duplicate_object` to allow safe re-runs. + +## Data Model + +### Users + +| Column | Type | Notes | +|--------|------|-------| +| `id` | int PK | | +| `username` | text UNIQUE NOT NULL | | +| `email` | text nullable | | +| `password_hash` | text nullable | NULL for OAuth-only accounts | +| `oauth_sub` | text UNIQUE nullable | OIDC subject identifier | +| `role` | text NOT NULL DEFAULT 'user' | First user auto-assigned 'admin' | +| `session_version` | int NOT NULL DEFAULT 1 | Bumped on password change to evict sessions | +| `created_at` | timestamptz | | + +### Notes (unified — includes tasks) + +| Column | Type | Notes | +|--------|------|-------| +| `id` | int PK | | +| `title` | text | | +| `body` | text | Markdown | +| `tags` | ARRAY[text] | GIN indexed; explicitly set by client | +| `parent_id` | int FK self nullable | Sub-tasks / sub-notes | +| `user_id` | int FK users nullable | CASCADE | +| `project_id` | int FK projects nullable | | +| `milestone_id` | int FK milestones nullable | | +| `status` | text nullable | `todo`/`in_progress`/`done` — non-null = task | +| `priority` | text nullable | `none`/`low`/`medium`/`high` | +| `due_date` | date nullable | | +| `created_at`, `updated_at` | timestamptz | | + +Indexes: GIN on `tags`, B-tree on `status`, B-tree on `title`. + +### Settings + +Composite PK `(user_id, key)`. Per-user key-value store. CRUD via `services/settings.py`. Used for: `default_model`, `assistant_name`, `briefing_enabled`, `briefing_locations`, `office_days`, etc. + +### Conversations / Messages + +`conversations`: `id`, `title`, `model`, `user_id`, `conversation_type` (`chat`/`briefing`/`mcp`), `briefing_date`, `rag_project_id` (nullable int — RAG scope: NULL=orphan-only, -1=all, positive=project), `created_at`, `updated_at`. +`messages`: `id`, `conversation_id` FK CASCADE, `role` (`user`/`assistant`), `content`, `status` (`done`/`generating`), `created_at`. + +Title auto-generated by LLM on first exchange, re-generated every 10th message. + +### Projects / Milestones + +`projects`: `id`, `user_id`, `title`, `description`, `goal`, `status` (`active`/`completed`/`archived`), `color`, `auto_summary` (nullable text — LLM-generated summary for `search_projects` scoring), `summary_updated_at` (nullable timestamptz), timestamps. +`milestones`: `id`, `user_id`, `project_id` FK CASCADE, `title`, `description`, `status`, `order_index`, timestamps. + +### Sharing & Access + +`project_shares`, `note_shares`: each has `shared_with_user_id` OR `shared_with_group_id` (exclusive), `permission` (`viewer`/`editor`/`admin`), `invited_by`. + +`groups`, `group_memberships`: platform-wide groups with `member`/`owner` roles. + +Permission resolution is centralised in `services/access.py`. `get_project_permission(uid, project_id)` checks ownership → direct share → group-based share → note→project inheritance, returning the highest applicable permission. + +### Briefing-Related Tables + +`rss_feeds`: `id`, `user_id`, `url`, `name`, `category`, `last_fetched_at`. +`rss_items`: `id`, `feed_id` FK, `guid`, `title`, `url`, `summary`, `pub_date`. +`weather_cache`: per-user cache with `lat`, `lon`, `location_name`, `forecast_json`, `fetched_at`. + +### API Keys + +`api_keys`: `id`, `user_id` FK CASCADE, `prefix` (first 8 chars, displayed in UI), `key_hash` (SHA-256 of full key — full key never stored), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at`. + +### App Logs + +`category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB. + +## Detailed File Reference + +### Backend (`src/fabledassistant/`) + +| File | Responsibility | +|------|---------------| +| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` | +| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper; accepts session cookie or `Authorization: Bearer ` | +| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods | +| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` | +| `models/note.py` | Unified Note model (notes + tasks) | +| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` | +| `models/conversation.py` | `Conversation` + `Message` models | +| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB | +| `models/api_key.py` | `ApiKey`: `id`, `user_id`, `prefix`, `key_hash` (SHA-256), `name`, `scope` (`read`/`write`), `created_at`, `last_used_at` | +| `models/event.py` | Internal events store (`Event` model: `id`, `user_id`, `title`, `description`, `start_dt`, `end_dt`, `all_day`, `location`, `caldav_uid` nullable, `color`, timestamps) | +| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) | +| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards | +| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) | +| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm; briefing conversation routes | +| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts | +| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) | +| `routes/task_logs.py` | Task work log CRUD (`GET/POST/DELETE /api/tasks/:id/logs`) | +| `routes/projects.py` | Projects CRUD + summary endpoint | +| `routes/milestones.py` | Milestones CRUD under `/api/projects/:id/milestones` | +| `routes/settings.py` | Per-user settings key-value (`GET/PUT /api/settings/:key`) | +| `routes/briefing.py` | Briefing conversation + reply + history; RSS feed management | +| `routes/groups.py` | Group CRUD + membership management (admin) | +| `routes/shares.py` | Share project/note with user or group; revoke; list incoming shares | +| `routes/in_app_notifications.py` | In-app notification list + mark-read + count | +| `routes/push.py` | Web Push subscription subscribe/unsubscribe; VAPID public key | +| `routes/users.py` | User profile; admin user list + delete | +| `routes/images.py` | Serve cached images at `/api/images/` | +| `routes/export.py` | `GET /api/export` — personal Markdown ZIP or JSON array download | +| `routes/api_keys.py` | API key CRUD (`GET/POST/DELETE /api/api-keys`) | +| `routes/fable_mcp_dist.py` | `GET /api/fable-mcp/info` + `GET /api/fable-mcp/download` — package distribution | +| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation | +| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search | +| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens | +| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` | +| `services/api_keys.py` | `generate_key()`, `create_api_key()`, `list_api_keys()`, `revoke_api_key()`, `lookup_key()` (SHA-256 hash lookup) | +| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard | +| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` | +| `services/intent.py` | `classify_intent()` — fast non-streaming LLM call; intent skip heuristic; `_PRIOR_WORK_REFS` fast-path | +| `services/tools.py` | All LLM tool definitions + `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` dispatcher; duplicate guards; `_resolve_project()` 4-step lookup; `search_projects` and `set_rag_scope` tools | +| `services/projects.py` | Project CRUD + `generate_project_summary()` (Ollama, fire-and-forget) + `backfill_project_summaries()` (startup) | +| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes(orphan_only=False)` (pgvector cosine similarity) | +| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) | +| `services/notes.py` | Note CRUD, wikilink resolution, backlink queries, tag management | +| `services/note_versions.py` | Version snapshot on save; restore-from-version; diff metadata | +| `services/note_drafts.py` | Per-user per-note draft persistence (AI Assist pending state) | +| `services/settings.py` | `get_setting()`, `set_setting()`, `get_all_settings()` — key-value per user | +| `services/tag_suggestions.py` | `/api/notes/suggest-tags` — LLM-generated tag suggestions for note body | +| `services/access.py` | Permission resolution for all shared resources | +| `services/sharing.py` | Create/revoke shares; list shares; `get_shared_with_me()` | +| `services/groups.py` | Group CRUD; membership management | +| `services/notifications.py` | Create/read/mark-read in-app notifications | +| `services/task_logs.py` | Append/list/delete work log entries on tasks | +| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → `GenerationBuffer` stream | +| `services/briefing_scheduler.py` | APScheduler `BackgroundScheduler`; slots with catch-up logic; async-safe via `asyncio.create_task` | +| `services/briefing_conversations.py` | Briefing conversation persistence and history queries | +| `services/briefing_profile.py` | Per-user profile note that the assistant updates over time | +| `services/research.py` | SearXNG research pipeline: 5 sub-queries → parallel fetch → synthesis; `search_images` for image category | +| `services/events.py` | Internal events CRUD: `list_events`, `create_event`, `update_event`, `delete_event`, `get_event`; source of truth for all event LLM tools | +| `routes/events.py` | `/api/events` — event CRUD routes | +| `services/caldav.py` | Optional CalDAV sync — user-configured external server; syncs to/from internal store via `caldav_uid` FK; `is_caldav_configured()` guards tool activation | +| `services/calendar_sync.py` | Dead code — Radicale sync service; was trialled and removed | +| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap | +| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) | +| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup | +| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) | +| `services/email.py` | SMTP email sending for password reset; reads `SMTP_*` env vars | +| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache | +| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items | +| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection | + +### Frontend (`frontend/src/`) + +| File | Responsibility | +|------|---------------| +| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount | +| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect | +| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth | +| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` | +| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` | +| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` | +| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` | +| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` | +| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` | +| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response | +| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; scope chip (RAG project scope pill above input); message queue; bulk delete | +| `views/CalendarView.vue` | `/calendar` — FullCalendar v6 month/week/day views; click to create event; click event to open EventSlideOver | +| `components/EventSlideOver.vue` | Reusable slide-over for event create/edit/delete; used in ToolCallCard, HomeView, CalendarView | +| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage | +| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel | +| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management | +| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage | +| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave | +| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks | +| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" | +| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" | +| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` | +| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration | +| `components/ShareDialog.vue` | `` modal; user tab + group tab; current shares list | +| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over | +| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave | +| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft | +| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards | +| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm | +| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state | +| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard | +| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` | +| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` | +| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete | + +## Key Services + +| Service | Responsibility | +|---------|---------------| +| `services/access.py` | Permission resolution for all shared resources | +| `services/llm.py` | `build_context()`, RAG injection, history summarisation | +| `services/generation_task.py` | SSE streaming, tool-call loop, GenerationBuffer management | +| `services/tools.py` | All LLM tool implementations (`create_note`, `search_notes`, `get_weather`, …) | +| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` | +| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → briefing output | +| `services/briefing_scheduler.py` | APScheduler integration, catch-up logic for missed slots | +| `services/backup.py` | Full and per-user backup export/restore (version 2 format) | +| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast fetch + DB cache | +| `services/rss.py` | feedparser-based fetch, per-feed DB cache, prune-to-100 | + +## Authentication + +**Session cookies** — `HttpOnly`, `SameSite=Lax`, optionally `Secure` (`SECURE_COOKIES` env var). Session includes `session_version`; mismatch with DB value (after password change) results in 401 and session clear. + +**Bearer token / API keys** — `Authorization: Bearer ` accepted by `_check_auth()` as an alternative to session cookies. The raw key is SHA-256 hashed and looked up via `services/api_keys.py`. Keys with `scope=read` are rejected on non-safe methods (`POST`/`PATCH`/`DELETE`). Used by Fable MCP and any external API consumers. + +**Local auth + OIDC/OAuth (PKCE)** — `services/oauth.py` handles discovery and `find_or_create_oauth_user`. On OAuth login: checks existing `oauth_sub` → matching email → creates new user. + +See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions. + +## LLM Pipeline Internals + +### Intent Routing + +Before the main model runs, a lightweight intent classifier (`services/intent.py`) runs concurrently with `build_context()`. It makes a fast non-streaming call using a smaller dedicated model (`OLLAMA_INTENT_MODEL`, default `qwen2.5:7b`) to determine if the message requires a tool call. + +**Skip heuristic** — Intent classification is skipped entirely for short messages (≤10 words) with no action/object keywords, saving 400–800ms on conversational replies. + +**Prior-work fast-path** — `_PRIOR_WORK_REFS` regex detects phrases like "research you did", "note you made", "using your research" and returns no-tool immediately, preventing `search_web` from firing when the user references existing notes. + +If a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk (TTFT), the tool executes, then the main model generates a follow-up with the tool result. For chat-only responses the main model streams directly. + +### Tool Loop + +Multi-round tool loop (max 5 rounds). All implementations in `services/tools.py`; `execute_tool(user_id, tool_name, arguments, conv_id=None, workspace_project_id=None)` is the dispatcher. `conv_id` and `workspace_project_id` are threaded in from `run_generation()` so tools like `set_rag_scope` can write to the current conversation. + +**Duplicate protection on `create_note` / `create_task`:** +1. Exact title match (case-insensitive) → hard block, redirect to `update_note` +2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block +3. Semantic content similarity (threshold 0.90, body ≥ 200 chars) → soft block with `requires_confirmation: true` + +**Project resolution** (`_resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55. + +### Context Window and Summarisation + +`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure. + +History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens. + +### Web Research Pipeline + +`services/research.py` implements a full autonomous research pipeline: +1. Intent model generates 5 focused sub-queries +2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter) +3. Up to 15 unique URLs fetched in parallel +4. Up to 12 sources passed to synthesis LLM +5. Result saved as a note with `tags=["research"]` + +SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests. + +### Image Cache + +`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/`. The user's browser never contacts the original image host. + +Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB). + +## RAG Pipeline + +1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call; accepts `orphan_only` and `project_id` scope flags. +2. Notes ≥ 0.60 similarity auto-injected into system prompt (up to 3, 800 chars each). +3. Notes 0.45–0.60 surfaced in chat sidebar as "Suggested" (user clicks to include). +4. Explicitly included notes delivered full-body. +5. `excluded_note_ids` prevents the current note from being injected as its own context. + +### RAG Scope (three-value system) + +`conversations.rag_project_id` controls which notes are eligible for retrieval: + +| Value | Behaviour | +|-------|-----------| +| `NULL` (default) | Orphan notes only — notes with `project_id IS NULL` | +| `-1` | All notes — opt-in to global search | +| Positive int | That project's notes only | + +`build_context()` derives `orphan_only` + `effective_project_id` from this value before calling both the semantic and keyword search paths. + +**Scope tools** — Two LLM tools let the model discover and switch scope mid-conversation: +- `search_projects` — SequenceMatcher scoring over title + description + `auto_summary`; returns top 5 matching projects. +- `set_rag_scope` — persists the new `rag_project_id` to DB immediately; blocked in workspace view; causes the SSE `done` event to include `new_rag_scope` + `new_rag_scope_label` so the frontend chip updates reactively. + +**Project summaries** — `generate_project_summary()` calls Ollama (fire-and-forget) and stores the result in `projects.auto_summary`. Triggered on project update and note saves (debounced 1h). `backfill_project_summaries()` runs at startup. + +Embedding model: `nomic-embed-text` via Ollama. Backfill runs 30s after startup (background task). diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..f8a76a8 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,155 @@ +# Configuration + +Configuration is via environment variables. The `docker-compose.yml` file sets defaults for local development; override them in a `.env` file (gitignored). + +## Environment Variables + +### Core + +| Variable | Default | Description | +|----------|---------|-------------| +| `DATABASE_URL` | `postgresql+asyncpg://fabled:fabled@db/fabledassistant` | PostgreSQL async connection string | +| `SECRET_KEY` | `dev-secret-change-me` | Session signing key — **change this in production** | +| `SECRET_KEY_FILE` | — | Path to a Docker secret file containing the key (alternative to `SECRET_KEY`) | +| `LOG_LEVEL` | `INFO` | Logging verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`) | +| `SECURE_COOKIES` | `false` | Set `true` when running behind TLS | +| `BASE_URL` | — | Public URL (e.g. `https://notes.example.com`) — required for OIDC redirect URIs and email links | + +### LLM / Ollama + +| Variable | Default | Description | +|----------|---------|-------------| +| `OLLAMA_URL` | `http://ollama:11434` | Ollama API base URL | +| `OLLAMA_MODEL` | `llama3.2` | Default LLM model (used as fallback; per-user setting overrides this) | +| `EMBEDDING_MODEL` | `nomic-embed-text` | Model used for semantic search / RAG embeddings | +| `OLLAMA_NUM_CTX` | `16384` | Context window size passed to Ollama for all generation calls | + +### Authentication / OIDC + +| Variable | Default | Description | +|----------|---------|-------------| +| `LOCAL_AUTH_ENABLED` | `true` | Set `false` to disable local username/password login (SSO-only mode) | +| `OIDC_ISSUER` | — | OIDC issuer URL (e.g. `https://auth.example.com/application/o/fabled/`) | +| `OIDC_CLIENT_ID` | — | OIDC client ID | +| `OIDC_CLIENT_SECRET` | — | OIDC client secret | +| `OIDC_CLIENT_SECRET_FILE` | — | Docker secret file alternative to `OIDC_CLIENT_SECRET` | +| `OIDC_SCOPES` | `openid profile email` | Space-separated OIDC scopes to request | + +See [sso-oauth.md](sso-oauth.md) for provider-specific setup. + +### Security / Proxy + +| Variable | Default | Description | +|----------|---------|-------------| +| `TRUST_PROXY_HEADERS` | `false` | Set `true` when behind a trusted reverse proxy to read real IP from `X-Forwarded-For` / `X-Real-IP` | + +### Web Search / Images + +| Variable | Default | Description | +|----------|---------|-------------| +| `SEARXNG_URL` | — | SearXNG base URL for web search and image search tools | +| `IMAGE_CACHE_DIR` | `/data/images` | Directory for cached search images | +| `IMAGE_MAX_BYTES` | `5242880` (5 MB) | Maximum size for cached images | + +### Data / Logging + +| Variable | Default | Description | +|----------|---------|-------------| +| `LOG_RETENTION_DAYS` | `90` | Days to keep app logs before automatic pruning | +| `DATA_DIR` | `/data` | Root directory for persistent data (VAPID keys, backups) | + +### Fable MCP Distribution + +| Variable | Default | Description | +|----------|---------|-------------| +| `FABLE_MCP_DIST_DIR` | `/app/dist` | Directory where the bundled `fable-mcp` wheel is placed at build time | + +## Docker Compose Setup + +### Development (`docker-compose.yml`) + +```bash +# Copy the example env file +cp .env.example .env +# Edit .env to set a real SECRET_KEY + +# Start the stack +docker compose up --build + +# App available at http://localhost:5000 +``` + +The first user to register becomes admin. Registration auto-closes after that (re-enable from Settings → Users). + +### Production (`docker-compose.prod.yml`) + +The production compose file adds: +- Docker Secrets for `SECRET_KEY_FILE` and `DATABASE_URL_FILE` +- Network isolation (internal bridge for DB + Ollama; only the app is externally accessible) +- Health checks and resource limits + +```bash +# Create Docker secrets +echo "$(python3 -c 'import secrets; print(secrets.token_hex(32))')" | docker secret create fabled_secret_key - +echo "postgresql+asyncpg://fabled:strongpassword@db/fabledassistant" | docker secret create fabled_db_url - + +# Deploy +docker stack deploy -c docker-compose.prod.yml fabled +``` + +## Production Deployment + +### Reverse Proxy (Required) + +Fabled Assistant does **not** handle SSL/TLS. Run it behind a reverse proxy: + +- **Nginx**, **Traefik**, or **Caddy** in front of the app container +- Terminate TLS at the proxy; forward to port 5000 +- **Do not expose port 5000 directly to the internet** +- Rate-limit auth endpoints: ≤ 5 req/min per IP on `/api/auth/login` and `/api/auth/register` + +Example Nginx location block: +```nginx +location / { + proxy_pass http://127.0.0.1:5000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Required for SSE streaming + proxy_buffering off; + proxy_read_timeout 600s; + proxy_send_timeout 600s; +} +``` + +If using `TRUST_PROXY_HEADERS=true`, ensure your proxy strips any client-supplied `X-Forwarded-For` headers before adding its own. + +### Security Checklist + +- **Strong `SECRET_KEY`** — generate with: + ```bash + python3 -c "import secrets; print(secrets.token_hex(32))" + ``` + Or use Docker Secrets via `SECRET_KEY_FILE`. +- **`SECURE_COOKIES=true`** — must be set when running behind TLS. +- **`BASE_URL`** — set to your public URL; required for OIDC redirects and email links. +- **Registration** — auto-closes after the first user (admin). Re-enable from Settings → Users or send invite links. +- **Session invalidation** — changing or resetting a password bumps `session_version`, evicting all other active sessions. Button also available in Settings → Account. +- **Keep Ollama on an internal network** — both compose files keep Ollama off the host network. Never expose the Ollama port publicly. +- **Default SECRET_KEY warning** — the app logs a `WARNING` on startup if the default dev key is in use. + +## Model Management + +Models are managed through Settings → General → Model Management in the web UI. You can: +- Pull new models by name (streams progress via SSE) +- View installed models with size and loaded/unloaded state +- Delete unused models + +The app auto-warms user-preferred models on startup (only models already installed — never auto-pulls). The embedding model (`nomic-embed-text`) is auto-pulled on startup if missing. + +Recommended models for 2× 8 GB GPU: +- **`qwen3:8b`** — strong reasoning + tools, fits in 8 GB, supports thinking mode +- **`qwen2.5:7b`** — fast, tool-capable, smaller context +- **`llama3.1:8b`** — reliable baseline, widely tested with tools diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..592ae86 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,173 @@ +# Development + +## Workflow + +All development is Docker-based. Do not install Python or Node dependencies locally. + +```bash +# Start the full stack (app + PostgreSQL + Ollama) +docker compose up --build + +# Rebuild after backend changes (frontend changes require rebuild too) +docker compose up --build app + +# Reset everything (wipes database) +docker compose down -v && docker compose up --build + +# Run checks (lint, format, typecheck, tests) +make check + +# Individual checks +make lint # ruff check src/ +make fmt # ruff format src/ +make typecheck # vue-tsc --noEmit +make test # pytest tests/ +``` + +## Frontend Hot Reload + +The Docker setup does not include Vite's hot-reload dev server. After frontend changes, rebuild the image. For faster iteration during active frontend work, you can run Vite locally: + +```bash +cd frontend +npm install +npm run dev # Vite dev server at http://localhost:5173 +``` + +Point the Vite dev server at the backend by setting `VITE_API_BASE_URL=http://localhost:5000` (or configure `vite.config.ts` proxy). + +## Database Migrations + +Alembic migrations run automatically on container startup (`alembic upgrade head` in the `CMD`). + +To create a new migration: +```bash +# Inside the running app container +docker compose exec app alembic revision -m "description_of_change" +# Edit the generated file in alembic/versions/ +``` + +Migration conventions: +- Use raw SQL with `IF NOT EXISTS` guards for idempotency +- Use `DO $$ BEGIN CREATE TYPE … EXCEPTION WHEN duplicate_object THEN NULL; END $$` for enum types +- Number migrations sequentially (e.g. `0027_add_something.py`) +- Always provide both `upgrade()` and `downgrade()` + +## CI/CD + +### Pipeline + +CI runs on Forgejo Actions with a custom runner base image (`py3.12-node22`): + +| Trigger | Jobs | Docker tags pushed | +|---------|------|--------------------| +| Push to `dev` | typecheck + lint + test → build | `:dev`, `:` | +| Tag `v*` on `main` | typecheck + lint + test → build | `:latest`, `:`, `:` | +| Push to `main` | typecheck + lint + test | (no build) | + +### Release Process + +1. Work on `dev` branch — CI validates on every push +2. When ready, open a PR from `dev` → `main` in Forgejo +3. Merge the PR +4. Create a release via the Forgejo UI on `main` with a `v*` tag (e.g. `v26.03.23.1` — CalVer: `YY.MM.DD.N`) +5. The tag push triggers CI → build job pushes `:latest` + `:` Docker images +6. After merging to main, sync dev back: + ```bash + git checkout dev && git merge main && git push origin dev + ``` + +### Custom Runner + +Runner base image: `infra/Dockerfile.runner-base` (Ubuntu 24.04 + Python 3.12 + Node 22 LTS). +Runner config: `infra/act-runner-config.yml` (label: `py3.12-node22`). +Runner compose: `infra/runner-compose.yml`. + +To activate a new runner registration, copy `infra/act-runner-config.yml` to the runner's config directory, delete the `.runner` registration file in the runner container, and restart the stack. + +### Docker Registry + +Images pushed to: `git.fabledsword.com/bvandeusen/fabledassistant` +Cache tag: `:cache` (reduces build time ~80%) + +Required secrets (repo → Settings → Secrets → Actions): +- `REGISTRY_USER` — Forgejo username +- `REGISTRY_TOKEN` — Forgejo PAT with `write:packages` scope + +## Migration Chain + +Current migration sequence (all idempotent raw SQL): + +``` +0001 create_notes_table +0002 create_tasks_table +0003 task_note_companion (data migration) +0004 merge_tasks_into_notes +0005 add_chat_tables +0006 add_settings_table +0007 add_title_and_updated_at_indexes +0008 add_users_and_user_id +0009 add_message_status +0010 add_app_logs_table +0011 add_password_reset_tokens +0012 add_invitation_tokens +0013 add_tool_calls_to_messages +0014 add_note_embeddings +0015 add_oauth_fields +0016 add_image_cache +0017 add_projects +0018 add_push_subscriptions +0019 add_events (dead code — internal CalDAV/Radicale table; Radicale was removed) +0020 add_milestones +0021 add_task_logs +0022 add_note_versions_and_drafts +0023 add_tags_to_note_versions +0024 add_session_version +0025 add_sharing_and_notifications +0026 add_briefing_tables +0027 add_api_keys +``` + +**Important:** Do NOT use `op.create_table()` or `sa.Enum()` — SQLAlchemy's event system can fire `CREATE TYPE` even with `create_type=False`, causing failures on re-run. Always use raw SQL with `IF NOT EXISTS` / `DO $$ BEGIN ... EXCEPTION WHEN duplicate_object` guards. + +## Project Conventions + +### Backend + +- Services: `async with async_session() as session:` — import from `fabledassistant.models` +- No `fabledassistant.database` module +- Blueprint per resource: `routes/notes.py`, `routes/tasks.py`, etc. +- All business logic in `services/`; routes are thin wrappers +- Permission checks via `services/access.py` — never inline ownership checks in routes + +### Frontend + +- API calls via `frontend/src/api/client.ts` typed helpers (`apiGet`, `apiPost`, `apiPatch`, `apiDelete`) +- Pinia stores for shared state; local `ref()` for component-only state +- Composables in `composables/` for reusable behaviour (autosave, keyboard nav, tag suggestions, …) +- Views are page-level components in `views/`; reusable UI in `components/` + +### Commit Style + +``` +type(scope): short description + +Longer body if needed. + +Co-Authored-By: Claude Sonnet 4.6 +``` + +Types: `feat`, `fix`, `refactor`, `docs`, `chore`, `test` +Scopes: feature area (e.g. `chat`, `briefing`, `fable-mcp`, `notes`) + +## Testing + +```bash +# Run all tests +make test + +# Run specific test file +docker compose exec app /opt/venv/bin/pytest tests/test_auth.py -v +``` + +Tests are in `tests/`. They run against a real PostgreSQL instance in CI (not mocked). Keep tests integration-style where possible — mock failures have historically masked real migration bugs. diff --git a/docs/features.md b/docs/features.md new file mode 100644 index 0000000..6c1ad9a --- /dev/null +++ b/docs/features.md @@ -0,0 +1,154 @@ +# Features + +## Notes + +Write in Markdown with a live-preview editor (Tiptap/ProseMirror). Headings, bold, italic, lists, code blocks, and task checklists render inline. A slash-command menu (`/`) inserts common blocks. + +**Wikilinks** — Link notes with `[[Title]]` or `[[Title|Display Text]]` syntax. Clicking a wikilink navigates to (or auto-creates) the referenced note. The editor suggests existing note titles as candidate links while typing `[[`. Backlinks appear in the note viewer sidebar. + +**Tags** — First-class `ARRAY[text]` column. Tag autocomplete in the editor sidebar suggests existing tags. Hierarchical tags (`project/webapp`) supported — filtering by `project` matches all `project/*` children. Tags are browsable via the knowledge graph. + +**Version history** — Every body edit snapshots a version (up to 20 per note). Browse and restore from the editor's History panel. Diff view shows changes against the current body. + +**AI writing assist** — Select a passage or work on the full document. Give an instruction ("make this more concise", "add examples"). The assistant streams a proposal; a diff view shows changes to accept or reject. Drafts persist across page loads. + +**Link suggestions** — The editor detects note titles appearing as plain text in the body and suggests converting them to wikilinks. + +## Tasks + +Tasks carry status (`todo` → `in_progress` → `done`), priority (`none`/`low`/`medium`/`high`), due date, milestone assignment, and a parent task (sub-tasks). + +**Task work logs** — Append progress log entries to a task with optional duration. Time tracking is visible in the task editor sidebar. + +**Sub-tasks** — Any task can have child tasks via `parent_id`. The task viewer shows sub-tasks inline. + +**Convert freely** — Convert a note to a task (sets `status=todo`) or a task back to a note from the viewer toolbar. + +## Projects and Milestones + +**Projects** — Group related notes and tasks. Each project has a title, description, goal, status (`active`/`completed`/`archived`), and a colour. + +**Milestones** — Ordered stages within a project. Tasks are assigned to milestones. Milestone completion percentage shown on the project page. + +**Kanban view** — `/projects/:id` groups tasks by milestone in a kanban-style column layout with status-advance buttons directly on cards (→ advance, ✓ complete). + +**Project Workspace** — `/workspace/:projectId` opens a three-panel environment (tasks / chat / notes) locked to a project. The AI assistant creates and updates content directly in the workspace; new notes auto-load in the editor and the task list refreshes automatically after tool calls. + +## Knowledge Graph + +`/graph` renders all notes, tasks, and tags as a D3 force-directed graph. Tag nodes cluster notes that share tags; invisible project hub nodes attract project members. Physics controls: repulsion, link distance, link strength, hub pull, gravity. Click any node to open a slide-in peek panel. Click a tag node to filter the notes list. + +## AI Chat + +Full conversation history with SSE streaming. Features: +- **RAG** — Semantically relevant notes (≥ 0.60 cosine similarity) auto-injected as context. Notes 0.45–0.60 shown in sidebar as "Suggested." +- **Attach notes** — Paperclip icon to include specific notes in context. +- **RAG scope chip** — Pill above the input bar shows the current note scope. Click to switch: "Orphan notes only" (default — project notes stay out of general chat), any active project, or "All notes." Scope is persisted per conversation. The AI can also call `search_projects` and `set_rag_scope` mid-conversation to switch scope automatically; the chip pulses when this happens. +- **Tool calls** — The assistant can create/update notes, tasks, projects, milestones, search the web, check weather, read RSS, query calendar events, and more. Tool calls display inline with confirm/deny for creates. +- **Thinking mode** — Toggle extended reasoning for complex questions. +- **Abort** — Stop button cancels in-flight generation. +- **Message queue** — Messages sent while generation is in progress are queued and drained sequentially. +- **Save to note** — Save any assistant reply directly as a note. +- **Bulk delete** — Select and delete multiple conversations. +- **Retention** — Conversations auto-pruned after configurable days (default 90). + +## Daily Briefing + +`/briefing` is a scheduled, dialogue-based morning briefing. The assistant compiles tasks, calendar events, projects, weather forecast, and RSS digest at configurable times, then checks in throughout the day. You can reply interactively. + +**Schedule** — Configurable slots: morning (default 4am compile), midday (8am check-in), evening (12pm check-in), night (4pm). Scheduler catches up missed slots on startup. + +**Configuration** — Settings → Briefing: enable toggle, location geocoding, office days, time slot toggles, RSS feed management, push notification toggle. + +**RSS feeds** — Add feed URLs with optional name and category. Feeds are fetched and cached; the briefing digest includes recent items. Category badges shown in the UI. Feeds can be manually refreshed. + +**Weather** — Location-based forecast via Open-Meteo. Multiple locations supported (home, work, or any city name). Geocoding via Nominatim. + +**Profile note** — The assistant maintains a profile note for each user that it updates based on briefing conversations, improving personalisation over time. + +## Web Research + +The assistant can search the web (SearXNG) and fetch pages, synthesising findings into notes. A lightweight `search_web` tool answers quick questions inline without saving. Requires `SEARXNG_URL` to be configured. + +## Calendar + +`/calendar` shows a full FullCalendar view (month, week, day). Click an empty slot to create an event; click an existing event to edit or delete it via a slide-over panel. + +**Internal events store** — Events are stored in the app database (`events` table), making them available without any external calendar. Fields: title, description, start/end datetime, all-day toggle, location, colour. + +**AI tools** — `create_event`, `list_events`, `search_events`, `update_event`, `delete_event` all operate on the internal store. Tool-call result cards in chat are clickable and open the same EventSlideOver for editing. + +**HomeView widget** — The dashboard shows today's and the next 7 days' events as clickable cards above the hero project. + +**CalDAV sync (optional)** — Connect an external CalDAV server (Nextcloud, Radicale, etc.) in Settings → Integrations. Events sync bidirectionally via a `caldav_uid` field. + +## Sharing and Collaboration + +**Share** — Share any project or note/task with users or groups at `viewer`/`editor`/`admin` permission levels. Share button in the viewer/project toolbar opens a dialog. + +**Groups** — Admins create platform-wide groups and assign users `member`/`owner` roles. Share a resource with a group in one action. + +**Shared with me** — `/shared` lists all incoming shared projects and notes with permission badges. + +**Notifications** — Bell icon in nav shows unread count (60s polling). Notifications generated for: project shared, note shared, added to group. Click navigates to the resource. + +**Push notifications** — Web Push (VAPID) notifies when AI generation completes, even in another tab. Works over HTTPS only. Configurable per-user. + +## Quick Capture + +Quick capture from the Android app routes to the intent classifier. It creates notes, tasks, or projects based on content — using the user's configured model, not the hardcoded default. + +## Data Export and Backup + +- **Personal export** — Settings → Data: download all notes/tasks as a Markdown ZIP (with YAML frontmatter) or JSON array. +- **Admin backup** — Full application backup (version 2): includes projects, milestones, task logs, AI drafts, note versions, push subscriptions. ID remapping on restore for cross-instance migration. + +## PWA + +Installable as a desktop or mobile app. Service worker caches the shell; push notifications are suppressed when the relevant tab is already focused. Works over HTTPS only in Firefox. + +## Settings + +Settings are tabbed: + +| Tab | Contents | +|-----|----------| +| General | Assistant name, default model, model management (pull/delete) | +| Account | Email change, password change, session invalidation | +| Notifications | Push notification subscription, briefing push toggle | +| Integrations | CalDAV configuration, SearXNG status | +| Data | Personal export, backup/restore (admin) | +| Briefing | Enable, location, office days, slots, RSS feeds, weather | +| API Keys | Create/revoke API keys, Fable MCP download and install | +| Config (admin) | Base URL, SMTP, OIDC settings | +| Users (admin) | User list, invite links, registration toggle | +| Logs (admin) | Error, audit, and usage logs with search | +| Groups (admin) | Create/manage groups and membership | + +## Roadmap + +- Email integration (read/send via IMAP/SMTP tools in chat) +- Session invalidation on user deletion +- Flutter push notifications (requires FCM/APNs — separate from web VAPID) +- Flutter milestone support in project view + +## Keyboard Shortcuts + +| Key | Action | +|-----|--------| +| `g` + `h` | Go to Home | +| `g` + `n` | Go to Notes | +| `g` + `t` | Go to Tasks | +| `g` + `p` | Go to Projects | +| `g` + `c` | Go to Chat | +| `g` (bare) | Go to Graph | +| `n` | New note | +| `t` | New task | +| `c` | Focus chat input | +| `e` | Edit current item | +| `/` | Search | +| `?` | Show shortcuts panel | +| `j` / `k` | Navigate list items | +| `Enter` | Open selected item | +| `Escape` | Close panel / blur / go home (progressive) | +| `Ctrl+S` | Save in editor | diff --git a/docs/plans/2026-03-25-briefing-improvements.md b/docs/plans/2026-03-25-briefing-improvements.md new file mode 100644 index 0000000..533e814 --- /dev/null +++ b/docs/plans/2026-03-25-briefing-improvements.md @@ -0,0 +1,2172 @@ +# Briefing Service Improvements — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the briefing's repetitive, prose-only output with a pre-filtered, structured pipeline that tracks task changes, classifies RSS topics, gates stale weather, and surfaces news source links with per-story reactions. + +**Architecture:** Pre-processing stage runs before gather — task change detection diffs against a DB snapshot, RSS filtering applies topic preferences and reaction weights, weather is gated at 24h. Structured card metadata is stored on `Message.metadata` (JSONB) at write time; the frontend reads it on load to render `WeatherCard.vue` and reaction buttons without any SSE changes. + +**Tech Stack:** Python 3.12 + SQLAlchemy 2.0 async, Quart, httpx (Open-Meteo `current_weather=true` + `past_days=1`), Vue 3 + TypeScript, existing `TagInput.vue`, FastMCP (fable-mcp). + +**Spec:** `docs/specs/2026-03-25-briefing-improvements-design.md` + +--- + +## File Map + +### New backend files +- `alembic/versions/0028_add_briefing_improvements.py` +- `src/fabledassistant/services/rss_classifier.py` +- `src/fabledassistant/services/briefing_preferences.py` + +### Modified backend files +- `src/fabledassistant/models/conversation.py` — add `Message.metadata` JSONB column + `to_dict()` +- `src/fabledassistant/models/rss_feed.py` — add `RssItem.topics`, `RssItem.classified_at` +- `src/fabledassistant/services/briefing_conversations.py` — `post_message(... metadata=None)` +- `src/fabledassistant/services/weather.py` — add `current_weather=true`, `past_days=1`, `parse_weather_card_data()` +- `src/fabledassistant/services/rss.py` — trigger classification after storing new items +- `src/fabledassistant/services/briefing_pipeline.py` — pre-processing stage, task snapshot helpers, return `(text, metadata)` from `run_compilation` +- `src/fabledassistant/routes/briefing.py` — unpack tuple from `run_compilation`; add reaction endpoints + +### New frontend files +- `frontend/src/components/WeatherCard.vue` + +### Modified frontend files +- `frontend/src/api/client.ts` — add `postRssReaction()`, `deleteRssReaction()` +- `frontend/src/views/BriefingView.vue` — render `WeatherCard`, reaction buttons from metadata +- `frontend/src/views/SettingsView.vue` — add "News Preferences" subsection + +### New MCP files +- `fable-mcp/fable_mcp/tools/briefing.py` + +### Modified MCP files +- `fable-mcp/fable_mcp/server.py` — register briefing tools + +### Test files +- `tests/test_briefing_pipeline.py` — extend with new tests +- `tests/test_weather_service.py` — extend with card parser tests +- `tests/test_rss_service.py` — extend with classifier + preferences tests + +--- + +## Task 1: Migration 0028 + +**Files:** +- Create: `alembic/versions/0028_add_briefing_improvements.py` + +- [ ] **Step 1: Create the migration file** + +```python +"""Add briefing improvements: rss_items topics/classified_at, messages metadata, + rss_item_reactions, briefing_task_snapshot.""" + +from alembic import op + +revision = "0028" +down_revision = "0027" + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE rss_items + ADD COLUMN IF NOT EXISTS topics TEXT[] DEFAULT '{}', + ADD COLUMN IF NOT EXISTS classified_at TIMESTAMPTZ + """) + op.execute(""" + ALTER TABLE messages + ADD COLUMN IF NOT EXISTS metadata JSONB + """) + op.execute(""" + CREATE TABLE IF NOT EXISTS rss_item_reactions ( + id SERIAL PRIMARY KEY, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + rss_item_id INTEGER NOT NULL REFERENCES rss_items(id) ON DELETE CASCADE, + reaction TEXT NOT NULL CHECK (reaction IN ('up', 'down')), + created_at TIMESTAMPTZ DEFAULT NOW(), + UNIQUE (user_id, rss_item_id) + ) + """) + op.execute( + "CREATE INDEX IF NOT EXISTS ix_rss_item_reactions_user_id " + "ON rss_item_reactions(user_id)" + ) + op.execute(""" + CREATE TABLE IF NOT EXISTS briefing_task_snapshot ( + id SERIAL PRIMARY KEY, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + task_id INTEGER NOT NULL REFERENCES notes(id) ON DELETE CASCADE, + snapshot_hash TEXT NOT NULL, + last_briefed TIMESTAMPTZ DEFAULT NOW(), + UNIQUE (user_id, task_id) + ) + """) + op.execute( + "CREATE INDEX IF NOT EXISTS ix_briefing_task_snapshot_user_id " + "ON briefing_task_snapshot(user_id)" + ) + + +def downgrade() -> None: + op.execute("DROP TABLE IF EXISTS briefing_task_snapshot") + op.execute("DROP TABLE IF EXISTS rss_item_reactions") + op.execute("ALTER TABLE messages DROP COLUMN IF EXISTS metadata") + op.execute("ALTER TABLE rss_items DROP COLUMN IF EXISTS classified_at") + op.execute("ALTER TABLE rss_items DROP COLUMN IF EXISTS topics") +``` + +- [ ] **Step 2: Apply migration inside the running container** + +```bash +docker compose exec app alembic upgrade head +``` + +Expected: `Running upgrade 0027 -> 0028, Add briefing improvements` + +- [ ] **Step 3: Commit** + +```bash +git add alembic/versions/0028_add_briefing_improvements.py +git commit -m "feat(briefing): add migration 0028 — briefing improvements schema" +``` + +--- + +## Task 2: Update SQLAlchemy Models + +**Files:** +- Modify: `src/fabledassistant/models/conversation.py` +- Modify: `src/fabledassistant/models/rss_feed.py` + +- [ ] **Step 1: Add `metadata` column to `Message` in `models/conversation.py`** + +In the `Message` class, after the existing `tool_calls` column, add: + +```python +metadata: Mapped[dict | None] = mapped_column(JSONB, nullable=True) +``` + +In `Message.to_dict()`, add `"metadata": self.metadata` to the returned dict. + +- [ ] **Step 2: Add `topics` and `classified_at` columns to `RssItem` in `models/rss_feed.py`** + +Add imports at the top (if not already present): +```python +from sqlalchemy import ARRAY, DateTime, Text # ARRAY is new +from sqlalchemy.dialects.postgresql import ARRAY as PG_ARRAY +``` + +Actually SQLAlchemy's native ARRAY works for PostgreSQL. Use: +```python +from sqlalchemy import ARRAY, Text +``` + +In the `RssItem` class, after `fetched_at`, add: + +```python +topics: Mapped[list[str]] = mapped_column( + ARRAY(Text), nullable=False, default=list, server_default="{}" +) +classified_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True +) +``` + +In `RssItem.to_dict()`, add: +```python +"topics": self.topics or [], +"classified_at": self.classified_at.isoformat() if self.classified_at else None, +``` + +- [ ] **Step 3: Rebuild the container to pick up model changes** + +```bash +docker compose up --build -d app +``` + +- [ ] **Step 4: Write a quick smoke test (no DB needed)** + +In `tests/test_briefing_models.py`, add: + +```python +def test_message_metadata_field_exists(): + from fabledassistant.models.conversation import Message + m = Message(conversation_id=1, role="assistant", content="hi", metadata={"foo": 1}) + assert m.metadata == {"foo": 1} + d = m.to_dict() + assert "metadata" in d + +def test_rss_item_topics_field_exists(): + from fabledassistant.models.rss_feed import RssItem + item = RssItem(feed_id=1, guid="x", title="Test", topics=["technology"]) + assert item.topics == ["technology"] + d = item.to_dict() + assert "topics" in d + assert "classified_at" in d +``` + +- [ ] **Step 5: Run tests** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_briefing_models.py -v +``` + +Expected: both new tests PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/fabledassistant/models/conversation.py src/fabledassistant/models/rss_feed.py tests/test_briefing_models.py +git commit -m "feat(briefing): add Message.metadata and RssItem.topics/classified_at columns" +``` + +--- + +## Task 3: Extend `post_message()` + +**Files:** +- Modify: `src/fabledassistant/services/briefing_conversations.py` + +- [ ] **Step 1: Write the failing test** + +In `tests/test_briefing_pipeline.py`, add: + +```python +@pytest.mark.asyncio +async def test_post_message_accepts_metadata(): + """post_message should accept an optional metadata dict and store it.""" + from unittest.mock import AsyncMock, patch, MagicMock + + mock_msg = MagicMock() + mock_msg.id = 1 + + with patch("fabledassistant.services.briefing_conversations.async_session") as mock_session_cls: + mock_session = AsyncMock() + mock_session.__aenter__ = AsyncMock(return_value=mock_session) + mock_session.__aexit__ = AsyncMock(return_value=False) + mock_session.add = MagicMock() + mock_session.get = AsyncMock(return_value=None) + mock_session.commit = AsyncMock() + mock_session.refresh = AsyncMock() + mock_session_cls.return_value = mock_session + + from fabledassistant.services.briefing_conversations import post_message + import importlib + import fabledassistant.services.briefing_conversations as bc + importlib.reload(bc) + + # Should not raise TypeError + await bc.post_message(1, "assistant", "text", metadata={"rss_item_ids": [1, 2]}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_briefing_pipeline.py::test_post_message_accepts_metadata -v +``` + +Expected: FAIL — `TypeError: post_message() got an unexpected keyword argument 'metadata'` + +- [ ] **Step 3: Update `post_message()` signature** + +In `services/briefing_conversations.py`, change: + +```python +async def post_message(conversation_id: int, role: str, content: str) -> Message: + """Append a message to a briefing conversation.""" + async with async_session() as session: + msg = Message( + conversation_id=conversation_id, + role=role, + content=content, + status="complete", + ) +``` + +To: + +```python +async def post_message( + conversation_id: int, + role: str, + content: str, + metadata: dict | None = None, +) -> Message: + """Append a message to a briefing conversation.""" + async with async_session() as session: + msg = Message( + conversation_id=conversation_id, + role=role, + content=content, + status="complete", + metadata=metadata, + ) +``` + +- [ ] **Step 4: Run test to verify it passes** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_briefing_pipeline.py::test_post_message_accepts_metadata -v +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/fabledassistant/services/briefing_conversations.py tests/test_briefing_pipeline.py +git commit -m "feat(briefing): extend post_message() with optional metadata parameter" +``` + +--- + +## Task 4: Weather Service — Card Data Parser + +**Files:** +- Modify: `src/fabledassistant/services/weather.py` +- Test: `tests/test_weather_service.py` + +- [ ] **Step 1: Write failing tests** + +Open `tests/test_weather_service.py` and add: + +```python +def test_parse_weather_card_data_returns_none_when_stale(): + """Should return None when cache is older than 24 hours.""" + from datetime import datetime, timezone, timedelta + from unittest.mock import MagicMock + from fabledassistant.services.weather import parse_weather_card_data + + cache = MagicMock() + cache.fetched_at = datetime.now(timezone.utc) - timedelta(hours=25) + cache.forecast_json = {} + assert parse_weather_card_data(cache) is None + + +def test_parse_weather_card_data_returns_card_schema(): + """Should return the correct metadata.weather schema for fresh cache.""" + from datetime import datetime, timezone, timedelta, date + from unittest.mock import MagicMock + from fabledassistant.services.weather import parse_weather_card_data + + today = date.today().isoformat() + yesterday = (date.today() - timedelta(days=1)).isoformat() + tomorrow = (date.today() + timedelta(days=1)).isoformat() + + cache = MagicMock() + cache.fetched_at = datetime.now(timezone.utc) + cache.location_label = "Berlin, DE" + cache.forecast_json = { + "current_weather": {"temperature": 12.0, "weathercode": 2}, + "daily": { + "time": [yesterday, today, tomorrow], + "temperature_2m_max": [14.0, 16.0, 18.0], + "temperature_2m_min": [9.0, 8.0, 10.0], + "precipitation_sum": [0.0, 0.0, 1.5], + "weathercode": [1, 2, 61], + "windspeed_10m_max": [10.0, 12.0, 8.0], + } + } + + card = parse_weather_card_data(cache) + assert card is not None + assert card["current_temp"] == 12 + assert card["condition"] == "Partly cloudy" + assert card["today_high"] == 16 + assert card["today_low"] == 8 + assert card["yesterday_high"] == 14 + assert card["yesterday_low"] == 9 + assert len(card["forecast"]) >= 1 + assert card["location"] == "Berlin, DE" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_weather_service.py -v -k "parse_weather_card" +``` + +Expected: FAIL — `ImportError: cannot import name 'parse_weather_card_data'` + +- [ ] **Step 3: Update `_fetch_open_meteo` to request current weather + past day** + +In `services/weather.py`, update `_fetch_open_meteo`: + +```python +async def _fetch_open_meteo(lat: float, lon: float) -> dict: + """Fetch 7-day forecast from Open-Meteo with current conditions and yesterday's data.""" + async with httpx.AsyncClient(timeout=15.0) as client: + resp = await client.get(OPEN_METEO_URL, params={ + "latitude": lat, + "longitude": lon, + "daily": OPEN_METEO_DAILY, + "current_weather": "true", + "past_days": 1, + "timezone": "auto", + "forecast_days": 7, + }) + resp.raise_for_status() + return resp.json() +``` + +- [ ] **Step 4: Add `parse_weather_card_data()` to `services/weather.py`** + +Add after `detect_changes()`: + +```python +def parse_weather_card_data( + cache_row, + temp_unit: str = "C", +) -> dict | None: + """ + Parse a WeatherCache row into the metadata.weather card schema. + Returns None if the cache is stale (older than 24 hours). + """ + from datetime import date, timedelta + + if cache_row is None or cache_row.fetched_at is None: + return None + + age_seconds = (datetime.now(timezone.utc) - cache_row.fetched_at).total_seconds() + if age_seconds > 86400: + return None + + raw = cache_row.forecast_json or {} + current_weather = raw.get("current_weather", {}) + days = parse_forecast(raw) + + today_str = date.today().isoformat() + yesterday_str = (date.today() - timedelta(days=1)).isoformat() + + today_day = next((d for d in days if d["date"] == today_str), None) + yesterday_day = next((d for d in days if d["date"] == yesterday_str), None) + future_days = [d for d in days if d["date"] > today_str][:5] + + def to_temp(c: float) -> int: + if temp_unit == "F": + return round(c * 9 / 5 + 32) + return round(c) + + def day_label(date_str: str) -> str: + from datetime import date as _date + try: + return _date.fromisoformat(date_str).strftime("%a") + except ValueError: + return date_str + + return { + "location": getattr(cache_row, "location_label", ""), + "fetched_at": cache_row.fetched_at.isoformat(), + "current_temp": to_temp(current_weather.get("temperature", 0)), + "condition": describe_weathercode(current_weather.get("weathercode", 0)), + "today_high": to_temp(today_day["temp_max"]) if today_day else None, + "today_low": to_temp(today_day["temp_min"]) if today_day else None, + "yesterday_high": to_temp(yesterday_day["temp_max"]) if yesterday_day else None, + "yesterday_low": to_temp(yesterday_day["temp_min"]) if yesterday_day else None, + "forecast": [ + { + "day": day_label(d["date"]), + "condition": d["description"], + "high": to_temp(d["temp_max"]), + "low": to_temp(d["temp_min"]), + } + for d in future_days + ], + } +``` + +- [ ] **Step 5: Run tests** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_weather_service.py -v -k "parse_weather_card" +``` + +Expected: both PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/fabledassistant/services/weather.py tests/test_weather_service.py +git commit -m "feat(briefing): add past_days/current_weather to Open-Meteo fetch; add parse_weather_card_data()" +``` + +--- + +## Task 5: RSS Classifier Service + +**Files:** +- Create: `src/fabledassistant/services/rss_classifier.py` +- Test: `tests/test_rss_service.py` + +- [ ] **Step 1: Write failing test** + +In `tests/test_rss_service.py`, add: + +```python +@pytest.mark.asyncio +async def test_classify_items_batch_returns_topic_map(): + """classify_items_batch should return a dict mapping item_id to topic list.""" + from unittest.mock import AsyncMock, patch + + fake_response = '{"1": ["technology", "ai"], "2": ["politics"]}' + + with patch( + "fabledassistant.services.rss_classifier._llm_classify", + new_callable=AsyncMock, + return_value=fake_response, + ): + from fabledassistant.services import rss_classifier + import importlib; importlib.reload(rss_classifier) + + items = [ + {"id": 1, "title": "OpenAI releases GPT-5", "content": "..."}, + {"id": 2, "title": "EU passes new law", "content": "..."}, + ] + result = await rss_classifier.classify_items_batch(items, user_include_topics=[]) + assert result[1] == ["technology", "ai"] + assert result[2] == ["politics"] + + +@pytest.mark.asyncio +async def test_classify_items_batch_handles_llm_failure(): + """classify_items_batch should return empty lists on LLM error.""" + from unittest.mock import AsyncMock, patch + + with patch( + "fabledassistant.services.rss_classifier._llm_classify", + new_callable=AsyncMock, + side_effect=Exception("LLM unavailable"), + ): + from fabledassistant.services import rss_classifier + import importlib; importlib.reload(rss_classifier) + + items = [{"id": 5, "title": "Some news", "content": ""}] + result = await rss_classifier.classify_items_batch(items, user_include_topics=[]) + assert result == {} # Empty on failure — items stay unclassified +``` + +- [ ] **Step 2: Run tests to confirm FAIL** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_rss_service.py -v -k "classify" +``` + +Expected: FAIL — module not found. + +- [ ] **Step 3: Create `services/rss_classifier.py`** + +```python +""" +RSS item topic classifier. + +Classifies RSS items into topic tags using a fast non-streaming LLM call. +Called from rss.py after new items are stored — fire-and-forget. +""" + +import json +import logging +from datetime import datetime, timezone + +import httpx + +from fabledassistant.config import Config + +logger = logging.getLogger(__name__) + +STANDARD_TOPICS = [ + "technology", "science", "politics", "business", + "health", "environment", "local", "entertainment", "sports", "other", +] + +_CLASSIFY_PROMPT = """\ +Classify each news item into 1-3 topics. Use only topics from this list: {vocab}. +Return ONLY a JSON object mapping item_id (as string) to a list of topics. +Example: {{"1": ["technology", "ai"], "2": ["politics"]}} + +Items: +{items_block}""" + + +async def _llm_classify(prompt: str, model: str) -> str: + """Make a fast non-streaming LLM call and return the raw text response.""" + payload = { + "model": model, + "messages": [{"role": "user", "content": prompt}], + "stream": False, + "options": {"num_ctx": 2048, "temperature": 0.0}, + } + async with httpx.AsyncClient(timeout=30.0) as client: + resp = await client.post(f"{Config.OLLAMA_URL}/api/chat", json=payload) + resp.raise_for_status() + return resp.json().get("message", {}).get("content", "") + + +async def classify_items_batch( + items: list[dict], + user_include_topics: list[str], + model: str | None = None, +) -> dict[int, list[str]]: + """ + Classify a batch of RSS items into topic tags. + + Args: + items: list of dicts with 'id', 'title', 'content' + user_include_topics: extra topics from user preferences to add to vocabulary + model: Ollama model name; defaults to Config.OLLAMA_MODEL + + Returns: + dict mapping item_id (int) -> list of topic strings. + Items not returned had classification fail; callers should leave classified_at=NULL. + """ + if not items: + return {} + + if model is None: + model = Config.OLLAMA_MODEL + + vocab = STANDARD_TOPICS + [t for t in user_include_topics if t not in STANDARD_TOPICS] + items_block = "\n".join( + f"[{item['id']}] {item['title']} — {item.get('content', '')[:300]}" + for item in items + ) + prompt = _CLASSIFY_PROMPT.format(vocab=", ".join(vocab), items_block=items_block) + + try: + raw = await _llm_classify(prompt, model) + # Extract JSON from response (LLM may wrap it in markdown) + raw = raw.strip() + if raw.startswith("```"): + raw = raw.split("```")[1] + if raw.startswith("json"): + raw = raw[4:] + parsed = json.loads(raw) + return {int(k): v for k, v in parsed.items() if isinstance(v, list)} + except Exception: + logger.warning("RSS classification failed", exc_info=True) + return {} + + +async def classify_and_store( + item_ids: list[int], + user_id: int, +) -> None: + """ + Classify unclassified RSS items and write results to DB. + Called as a fire-and-forget task from rss.py. + """ + from sqlalchemy import select + from fabledassistant.models import async_session + from fabledassistant.models.rss_feed import RssItem + from fabledassistant.services.settings import get_setting + + if not item_ids: + return + + # Load the items + async with async_session() as session: + result = await session.execute( + select(RssItem).where(RssItem.id.in_(item_ids)) + ) + items = list(result.scalars().all()) + + if not items: + return + + # Get user's include topics to extend vocabulary + raw_include = await get_setting(user_id, "briefing_include_topics", "[]") + try: + include_topics = json.loads(raw_include) if isinstance(raw_include, str) else [] + except Exception: + include_topics = [] + + model = await get_setting(user_id, "default_model", Config.OLLAMA_MODEL) + + # Classify in batches of 10 + batch_size = 10 + all_results: dict[int, list[str]] = {} + for i in range(0, len(items), batch_size): + batch = items[i: i + batch_size] + batch_dicts = [{"id": it.id, "title": it.title, "content": it.content} for it in batch] + results = await classify_items_batch(batch_dicts, include_topics, model=model) + all_results.update(results) + + # Write back to DB + now = datetime.now(timezone.utc) + async with async_session() as session: + for item in items: + item_db = await session.get(RssItem, item.id) + if item_db is None: + continue + topics = all_results.get(item.id) + if topics is not None: + item_db.topics = topics + item_db.classified_at = now + await session.commit() +``` + +- [ ] **Step 4: Run tests** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_rss_service.py -v -k "classify" +``` + +Expected: both PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/fabledassistant/services/rss_classifier.py tests/test_rss_service.py +git commit -m "feat(briefing): add rss_classifier service for LLM-based topic tagging" +``` + +--- + +## Task 6: Briefing Preferences Service + +> **Dependency:** Task 2 Step 2 must be complete before this task — `score_and_filter_items` expects items to have a `topics` key in their dicts, which comes from the `RssItem.topics` column added in Task 2. Run Task 2 first. + +**Files:** +- Create: `src/fabledassistant/services/briefing_preferences.py` +- Test: `tests/test_rss_service.py` + +- [ ] **Step 1: Write failing tests** + +In `tests/test_rss_service.py`, add: + +```python +def test_score_rss_items_excludes_blacklisted_topics(): + """Items with excluded topics should be removed.""" + from fabledassistant.services.briefing_preferences import score_and_filter_items + + items = [ + {"id": 1, "title": "Tech news", "topics": ["technology"], "published_at": "2026-03-25T08:00:00"}, + {"id": 2, "title": "Sports score", "topics": ["sports"], "published_at": "2026-03-25T08:00:00"}, + ] + result = score_and_filter_items( + items, + include_topics=["technology"], + exclude_topics=["sports"], + topic_scores={}, + max_items=10, + ) + ids = [r["id"] for r in result] + assert 1 in ids + assert 2 not in ids + + +def test_score_rss_items_boosts_included_topics(): + """Items matching include_topics should rank higher than neutral items.""" + from fabledassistant.services.briefing_preferences import score_and_filter_items + + items = [ + {"id": 1, "title": "Random news", "topics": ["other"], "published_at": "2026-03-25T07:00:00"}, + {"id": 2, "title": "Tech news", "topics": ["technology"], "published_at": "2026-03-25T06:00:00"}, + ] + result = score_and_filter_items( + items, + include_topics=["technology"], + exclude_topics=[], + topic_scores={}, + max_items=10, + ) + # Tech item should be ranked first despite being older + assert result[0]["id"] == 2 + + +def test_score_rss_items_no_preferences_returns_all(): + """With no preferences, all items should be returned sorted by recency.""" + from fabledassistant.services.briefing_preferences import score_and_filter_items + + items = [ + {"id": 1, "title": "A", "topics": [], "published_at": "2026-03-24T10:00:00"}, + {"id": 2, "title": "B", "topics": [], "published_at": "2026-03-25T10:00:00"}, + ] + result = score_and_filter_items(items, [], [], {}, max_items=10) + assert result[0]["id"] == 2 # Newer first +``` + +- [ ] **Step 2: Run tests to confirm FAIL** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_rss_service.py -v -k "score_rss" +``` + +Expected: FAIL — module not found. + +- [ ] **Step 3: Create `services/briefing_preferences.py`** + +```python +""" +Briefing preferences: load topic settings, aggregate reaction scores, +filter and rank RSS items for briefing inclusion. +""" + +import json +import logging +from datetime import datetime, timezone + +from sqlalchemy import func, select + +from fabledassistant.models import async_session + +logger = logging.getLogger(__name__) + + +async def load_topic_preferences(user_id: int) -> tuple[list[str], list[str]]: + """ + Return (include_topics, exclude_topics) from user settings. + """ + from fabledassistant.services.settings import get_setting + + raw_include = await get_setting(user_id, "briefing_include_topics", "[]") + raw_exclude = await get_setting(user_id, "briefing_exclude_topics", "[]") + + def _parse(raw) -> list[str]: + try: + val = json.loads(raw) if isinstance(raw, str) else raw + return [str(t) for t in val] if isinstance(val, list) else [] + except Exception: + return [] + + return _parse(raw_include), _parse(raw_exclude) + + +async def load_topic_reaction_scores(user_id: int) -> dict[str, float]: + """ + Aggregate per-topic reaction scores from the last 30 days. + Returns a dict of topic -> net_score (positive = liked, negative = disliked). + Uses rss_item_reactions joined to rss_items.topics. + """ + from fabledassistant.models.rss_feed import RssItem + + try: + async with async_session() as session: + # Raw SQL is simpler here due to ARRAY unnest + result = await session.execute( + __import__("sqlalchemy", fromlist=["text"]).text(""" + SELECT unnest(i.topics) AS topic, + SUM(CASE r.reaction WHEN 'up' THEN 1 ELSE -1 END) AS score + FROM rss_item_reactions r + JOIN rss_items i ON i.id = r.rss_item_id + WHERE r.user_id = :uid + AND r.created_at > NOW() - INTERVAL '30 days' + GROUP BY topic + """).bindparams(uid=user_id) + ) + return {row.topic: float(row.score) for row in result} + except Exception: + logger.warning("Failed to load topic reaction scores", exc_info=True) + return {} + + +def score_and_filter_items( + items: list[dict], + include_topics: list[str], + exclude_topics: list[str], + topic_scores: dict[str, float], + max_items: int = 10, +) -> list[dict]: + """ + Score, filter, and rank RSS items for briefing inclusion. + + Scoring: + - Hard-exclude: any item tagged with an excluded topic is removed. + - Base score: 0.0 + - +2.0 per topic that appears in include_topics + - +1.0 / -1.0 per topic based on reaction score (clamped per topic) + - Tiebreak: newer published_at wins + + Returns up to max_items items, highest score first. + Items with classified_at=None (unclassified) pass through with score=0. + """ + include_set = set(include_topics) + exclude_set = set(exclude_topics) + scored = [] + + for item in items: + item_topics = item.get("topics") or [] + + # Hard exclude + if exclude_set and any(t in exclude_set for t in item_topics): + continue + + score = 0.0 + for topic in item_topics: + if topic in include_set: + score += 2.0 + if topic in topic_scores: + score += max(-1.0, min(1.0, topic_scores[topic])) + + # Parse published_at for tiebreak + pub_str = item.get("published_at") or "" + try: + pub_ts = datetime.fromisoformat(pub_str).timestamp() if pub_str else 0.0 + except ValueError: + pub_ts = 0.0 + + scored.append((score, pub_ts, item)) + + # Sort: highest score first, then newest first + scored.sort(key=lambda x: (x[0], x[1]), reverse=True) + return [item for _, _, item in scored[:max_items]] +``` + +- [ ] **Step 4: Run tests** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_rss_service.py -v -k "score_rss" +``` + +Expected: all PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/fabledassistant/services/briefing_preferences.py tests/test_rss_service.py +git commit -m "feat(briefing): add briefing_preferences service for RSS scoring and filtering" +``` + +--- + +## Task 7: Task Change Detection + +**Files:** +- Modify: `src/fabledassistant/services/briefing_pipeline.py` +- Test: `tests/test_briefing_pipeline.py` + +- [ ] **Step 1: Write failing tests** + +In `tests/test_briefing_pipeline.py`, add: + +```python +def test_compute_task_snapshot_hash(): + """compute_task_hash should return a stable SHA-256 hex string.""" + from fabledassistant.services.briefing_pipeline import compute_task_hash + task = {"status": "todo", "priority": "high", "due_date": "2026-03-25", "title": "Write spec"} + h = compute_task_hash(task) + assert len(h) == 64 # SHA-256 hex + # Same inputs produce same hash + assert h == compute_task_hash(task) + # Different status produces different hash + assert h != compute_task_hash({**task, "status": "done"}) + + +@pytest.mark.asyncio +async def test_split_changed_tasks_all_new(): + """split_changed_tasks should return all tasks as changed when no snapshot exists.""" + from unittest.mock import AsyncMock, patch, MagicMock + from fabledassistant.services.briefing_pipeline import split_changed_tasks + + tasks = [ + {"task_id": 1, "title": "A", "status": "todo", "priority": "none", "due_date": None}, + ] + + with patch( + "fabledassistant.services.briefing_pipeline.async_session" + ) as mock_cls: + mock_session = AsyncMock() + mock_session.__aenter__ = AsyncMock(return_value=mock_session) + mock_session.__aexit__ = AsyncMock(return_value=False) + mock_session.execute = AsyncMock(return_value=MagicMock( + scalars=MagicMock(return_value=MagicMock(all=MagicMock(return_value=[]))) + )) + mock_cls.return_value = mock_session + + changed, unchanged_count = await split_changed_tasks(user_id=1, tasks=tasks) + assert len(changed) == 1 + assert unchanged_count == 0 +``` + +- [ ] **Step 2: Run tests to verify FAIL** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_briefing_pipeline.py -v -k "task_snapshot or task_hash or split_changed" +``` + +Expected: FAIL. + +- [ ] **Step 3: Add helpers to `briefing_pipeline.py`** + +Add these imports at the top of `briefing_pipeline.py`: +```python +import hashlib +from datetime import datetime, timezone +``` + +Add these functions after `format_task()`: + +```python +def compute_task_hash(task: dict) -> str: + """Stable SHA-256 of the task's key change-detectable fields.""" + key = "|".join([ + str(task.get("status") or ""), + str(task.get("priority") or ""), + str(task.get("due_date") or ""), + str(task.get("title") or ""), + ]) + return hashlib.sha256(key.encode()).hexdigest() + + +async def split_changed_tasks( + user_id: int, + tasks: list[dict], +) -> tuple[list[dict], int]: + """ + Compare tasks against the briefing_task_snapshot table. + Returns (changed_tasks, unchanged_count). + changed_tasks includes new tasks (no snapshot row) and tasks whose hash differs. + """ + from sqlalchemy import select, text + from fabledassistant.models import async_session + + if not tasks: + return [], 0 + + task_ids = [t["task_id"] for t in tasks if t.get("task_id")] + + async with async_session() as session: + result = await session.execute( + text(""" + SELECT task_id, snapshot_hash + FROM briefing_task_snapshot + WHERE user_id = :uid AND task_id = ANY(:ids) + """).bindparams(uid=user_id, ids=task_ids) + ) + snapshots = {row.task_id: row.snapshot_hash for row in result} + + changed = [] + unchanged_count = 0 + for task in tasks: + current_hash = compute_task_hash(task) + stored_hash = snapshots.get(task.get("task_id")) + if stored_hash is None or stored_hash != current_hash: + changed.append(task) + else: + unchanged_count += 1 + + return changed, unchanged_count + + +async def upsert_task_snapshots(user_id: int, tasks: list[dict]) -> None: + """Upsert snapshot hashes for all tasks included in this briefing.""" + from sqlalchemy import text + from fabledassistant.models import async_session + + if not tasks: + return + + now = datetime.now(timezone.utc) + async with async_session() as session: + for task in tasks: + task_id = task.get("task_id") + if not task_id: + continue + await session.execute( + text(""" + INSERT INTO briefing_task_snapshot (user_id, task_id, snapshot_hash, last_briefed) + VALUES (:uid, :tid, :hash, :now) + ON CONFLICT (user_id, task_id) + DO UPDATE SET snapshot_hash = EXCLUDED.snapshot_hash, + last_briefed = EXCLUDED.last_briefed + """).bindparams( + uid=user_id, + tid=task_id, + hash=compute_task_hash(task), + now=now, + ) + ) + await session.commit() +``` + +- [ ] **Step 4: Update `_gather_internal` to include `task_id`** + +In `_gather_internal`, change the task serialisation from: + +```python +all_tasks = [ + { + "title": t.title, + "status": t.status, + "due_date": t.due_date.isoformat() if t.due_date else None, + "priority": t.priority, + } + for t in all_task_objs +] +``` + +To: + +```python +all_tasks = [ + { + "task_id": t.id, + "title": t.title, + "status": t.status, + "due_date": t.due_date.isoformat() if t.due_date else None, + "priority": t.priority, + } + for t in all_task_objs +] +``` + +Also add `"all_tasks_raw": all_tasks` to the dict that `_gather_internal` returns, alongside the existing keys (`overdue_tasks`, `due_today`, etc.). This is the key `run_compilation` will use for snapshot diffing. + +- [ ] **Step 5: Run tests** + +```bash +docker compose exec app /opt/venv/bin/pytest tests/test_briefing_pipeline.py -v -k "task_snapshot or task_hash or split_changed" +``` + +Expected: all PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/fabledassistant/services/briefing_pipeline.py tests/test_briefing_pipeline.py +git commit -m "feat(briefing): add task change detection helpers and task_id to _gather_internal" +``` + +--- + +## Task 8: Wire Pre-processing into `run_compilation` + +**Files:** +- Modify: `src/fabledassistant/services/briefing_pipeline.py` +- Modify: `src/fabledassistant/routes/briefing.py` +- Modify: `src/fabledassistant/services/briefing_scheduler.py` (read the file first to find all `run_compilation` calls) + +- [ ] **Step 1: Read the briefing scheduler to find callers** + +```bash +docker compose exec app grep -n "run_compilation\|post_message" src/fabledassistant/services/briefing_scheduler.py +``` + +Note every line number that calls `run_compilation` or `post_message` — you must update all of them in step 4. + +- [ ] **Step 2: Update `_external_system_prompt` and `_external_user_prompt` for news cards** + +In `_external_system_prompt()`, replace the current string with: + +```python +def _external_system_prompt() -> str: + return ( + "You are a briefing assistant for external information. Your job is to present " + "selected news items and summarise any remaining RSS content. " + "IMPORTANT: Weather is handled separately — do NOT include any weather section.\n\n" + "Format each news item EXACTLY as:\n" + "**[Headline text](source_url)**\n" + "*Outlet Name · Day Month*\n" + "One or two sentence summary.\n\n" + "Present news items in the EXACT ORDER they are provided. Do not reorder them. " + "After the news cards, add a brief paragraph for any remaining context." + ) +``` + +- [ ] **Step 3: Rewrite `run_compilation` to add pre-processing and return `(text, metadata)`** + +Replace the existing `run_compilation` function entirely: + +```python +async def run_compilation( + user_id: int, + slot: str, + model: str | None = None, +) -> tuple[str, dict]: + """ + Run the full two-lane briefing pipeline for a user and slot. + Returns (briefing_text, metadata_dict) where metadata contains + weather card data and rss_item_ids for frontend rendering. + """ + if model is None: + model = await get_setting(user_id, "default_model", Config.OLLAMA_MODEL) + + from fabledassistant.services.briefing_profile import get_profile_body + from fabledassistant.services.briefing_preferences import ( + load_topic_preferences, + load_topic_reaction_scores, + score_and_filter_items, + ) + from fabledassistant.services.weather import parse_weather_card_data + + profile_body, temp_unit = await asyncio.gather( + get_profile_body(user_id), + _get_temp_unit(user_id), + ) + + # ── Pre-processing ────────────────────────────────────────────────────── + include_topics, exclude_topics = await load_topic_preferences(user_id) + topic_scores = await load_topic_reaction_scores(user_id) + + from fabledassistant.services.weather import get_cached_weather_rows + + # Parallel raw gather — include weather rows in the same gather to avoid a second DB round-trip + internal_data, external_data, weather_rows = await asyncio.gather( + _gather_internal(user_id), + _gather_external(user_id), + get_cached_weather_rows(user_id), + ) + + # Task change detection — uses the raw task dicts added in Task 7 + all_tasks = internal_data.get("all_tasks_raw", []) + changed_tasks, unchanged_count = await split_changed_tasks(user_id, all_tasks) + + # RSS filtering + raw_rss = external_data.get("rss_items") or [] + filtered_rss = score_and_filter_items( + raw_rss, + include_topics=include_topics, + exclude_topics=exclude_topics, + topic_scores=topic_scores, + max_items=10, + ) + rss_item_ids = [item["id"] for item in filtered_rss if item.get("id")] + + # Weather staleness gate — parse_weather_card_data returns None if data is >24h old + weather_card = parse_weather_card_data(weather_rows[0], temp_unit) if weather_rows else None + + # ── LLM Synthesis ────────────────────────────────────────────────────── + # Rebuild internal_data with changed tasks only + internal_data_filtered = dict(internal_data) + internal_data_filtered["unchanged_task_count"] = unchanged_count + # Replace task lists with only changed tasks (formatted) + today = internal_data["date"] + changed_formatted = [format_task(t) for t in changed_tasks] + internal_data_filtered["overdue_tasks"] = [ + f for f in changed_formatted + if any(t.get("due_date") and t["due_date"] < today and t.get("status") != "done" + for t in changed_tasks if format_task(t) == f) + ] + # Simplified: pass all changed tasks, let the LLM sort by urgency + internal_data_filtered["changed_tasks"] = changed_formatted + + # Build filtered external data (no weather — card handles it) + external_data_filtered = { + "rss_items": filtered_rss, + "weather": [], # Suppressed — handled by WeatherCard + } + + internal_text, external_text = await asyncio.gather( + _llm_synthesise( + _internal_system_prompt(profile_body), + _internal_user_prompt(internal_data_filtered, slot), + model, + ), + _llm_synthesise( + _external_system_prompt(), + _external_user_prompt(external_data_filtered, slot, temp_unit), + model, + ), + ) + + # ── Post-processing ──────────────────────────────────────────────────── + # Upsert task snapshots so next run can diff + await upsert_task_snapshots(user_id, all_tasks) + + # Build metadata + metadata: dict = {"rss_item_ids": rss_item_ids, "weather": weather_card} + + # Build output text + if not internal_text and not external_text: + logger.warning( + "Briefing compilation produced no content for user %d slot %s", user_id, slot + ) + return "", metadata + + greeting = slot_greeting(slot) + parts = [f"**{greeting} — {today}**", ""] + if internal_text: + parts += ["## Your Day", "", internal_text, ""] + if external_text: + parts += ["## The World", "", external_text] + + return "\n".join(parts).strip(), metadata +``` + +> **Note:** `_gather_internal` needs to also return the raw task objects (with `task_id`) for snapshot diffing. See step below. + +- [ ] **Step 4: Add `get_cached_weather_rows()` to `services/weather.py`** + +The new pipeline needs the raw `WeatherCache` ORM rows (not the processed dicts) so it can call `parse_weather_card_data()`. Add this function to `weather.py`: + +```python +async def get_cached_weather_rows(user_id: int) -> list: + """Return raw WeatherCache ORM rows for a user (for card parsing).""" + async with async_session() as session: + result = await session.execute( + select(WeatherCache).where(WeatherCache.user_id == user_id) + ) + return list(result.scalars().all()) +``` + +- [ ] **Step 5: Update `_internal_user_prompt` to handle changed tasks** + +The prompt currently references `overdue_tasks`, `due_today`, `high_priority`. Add an `unchanged_task_count` line: + +```python +def _internal_user_prompt(data: dict, slot: str) -> str: + lines = [f"Briefing slot: {slot}", f"Date: {data['date']}", ""] + if data.get("unchanged_task_count", 0) > 0: + lines.append( + f"({data['unchanged_task_count']} tasks are unchanged since the last briefing " + "— acknowledge briefly, do not list them.)" + ) + lines.append("") + # ... rest unchanged +``` + +- [ ] **Step 6: Update `routes/briefing.py` trigger endpoint** + +In `manual_trigger()`, unpack the tuple: + +```python +text, metadata = await run_compilation(g.user.id, slot, model) +msg = await post_message(conv.id, "assistant", text, metadata=metadata) +``` + +- [ ] **Step 7: Update `services/briefing_scheduler.py`** + +Find all calls to `run_compilation` (from step 1) and unpack the returned tuple, passing `metadata` to `post_message`. The pattern will be the same as step 6. + +- [ ] **Step 8: Run the full test suite** + +```bash +make test +``` + +Expected: all existing tests still PASS (no regressions). + +- [ ] **Step 9: Commit** + +```bash +git add src/fabledassistant/services/briefing_pipeline.py \ + src/fabledassistant/services/weather.py \ + src/fabledassistant/routes/briefing.py \ + src/fabledassistant/services/briefing_scheduler.py +git commit -m "feat(briefing): wire pre-processing pipeline; run_compilation now returns (text, metadata)" +``` + +--- + +## Task 9: Trigger RSS Classification from `rss.py` + +**Files:** +- Modify: `src/fabledassistant/services/rss.py` + +- [ ] **Step 1: Find the `fetch_and_cache_feed` function and add classification trigger** + +After `await _prune_old_items(feed_id)` (the last line of `fetch_and_cache_feed`), add: + +```python + # Queue classification for newly stored items if any were added + if new_count > 0 and new_item_ids: + import asyncio as _asyncio + from fabledassistant.services.rss_classifier import classify_and_store + # Fire-and-forget — don't await, don't block feed fetch + _asyncio.create_task(classify_and_store(new_item_ids, _feed_user_id)) + + return new_count +``` + +To make this work, you need to: +1. Collect `new_item_ids` during the loop (after `session.commit()`, refresh the items to get their DB-assigned IDs, or collect IDs inline) +2. Expose `_feed_user_id` by fetching it from the `RssFeed` row + +Update `fetch_and_cache_feed` to capture new item IDs and the feed's `user_id`: + +```python +async def fetch_and_cache_feed(feed_id: int, url: str) -> int: + # ... existing fetch/parse code ... + new_count = 0 + new_item_ids: list[int] = [] + feed_user_id: int | None = None + + async with async_session() as session: + for entry in parsed.entries: + # ... existing upsert code ... + item = RssItem(feed_id=feed_id, **item_data) + session.add(item) + new_count += 1 + + feed_row = await session.get(RssFeed, feed_id) + if feed_row: + feed_row.last_fetched_at = datetime.now(timezone.utc) + feed_user_id = feed_row.user_id + if not feed_row.title and parsed.feed.get("title"): + feed_row.title = parsed.feed.title[:200] + + await session.commit() + + # Collect IDs of newly inserted items after commit. + # We query classified_at IS NULL (not just the items inserted above) because + # classification is best-effort and may have failed on previous fetches. + # Re-queuing all unclassified items for this feed on each fetch is intentional: + # it provides automatic retry without a separate retry loop. The classifier + # only writes to items it successfully classifies, so already-classified items + # are not re-processed (they have classified_at set). + if new_count > 0: + result = await session.execute( + select(RssItem.id).where( + RssItem.feed_id == feed_id, + RssItem.classified_at.is_(None), + ) + ) + new_item_ids = list(result.scalars().all()) + + await _prune_old_items(feed_id) + + if new_count > 0 and new_item_ids and feed_user_id is not None: + import asyncio as _asyncio + from fabledassistant.services.rss_classifier import classify_and_store + _asyncio.create_task(classify_and_store(new_item_ids, feed_user_id)) + + return new_count +``` + +- [ ] **Step 2: Run the test suite** + +```bash +make test +``` + +Expected: all PASS (no regressions — classification is fire-and-forget so existing tests are unaffected). + +- [ ] **Step 3: Commit** + +```bash +git add src/fabledassistant/services/rss.py +git commit -m "feat(briefing): trigger RSS classification after new items are stored" +``` + +--- + +## Task 10: Reaction Endpoints + +**Files:** +- Modify: `src/fabledassistant/routes/briefing.py` + +- [ ] **Step 1: Add reaction endpoints at the bottom of `routes/briefing.py`** + +```python +# ── RSS Reactions ───────────────────────────────────────────────────────────── + +@briefing_bp.route("/rss-reactions", methods=["POST"]) +@_REQUIRE +async def upsert_rss_reaction(): + """Upsert a 👍/👎 reaction on an RSS item. Same reaction toggles off; opposite flips.""" + data = await request.get_json() + rss_item_id = data.get("rss_item_id") + reaction = data.get("reaction") + + if not rss_item_id or reaction not in ("up", "down"): + return jsonify({"error": "rss_item_id and reaction ('up'|'down') required"}), 400 + + from sqlalchemy import text as _text + + async with async_session() as session: + # Ownership check: verify item belongs to a feed owned by this user + result = await session.execute( + _text(""" + SELECT i.id FROM rss_items i + JOIN rss_feeds f ON f.id = i.feed_id + WHERE i.id = :item_id AND f.user_id = :uid + """).bindparams(item_id=rss_item_id, uid=g.user.id) + ) + if result.first() is None: + return jsonify({"error": "Not found"}), 404 + + # Check existing reaction + existing = await session.execute( + _text(""" + SELECT id, reaction FROM rss_item_reactions + WHERE user_id = :uid AND rss_item_id = :item_id + """).bindparams(uid=g.user.id, item_id=rss_item_id) + ) + row = existing.first() + + if row is None: + # Insert + await session.execute( + _text(""" + INSERT INTO rss_item_reactions (user_id, rss_item_id, reaction) + VALUES (:uid, :item_id, :reaction) + """).bindparams(uid=g.user.id, item_id=rss_item_id, reaction=reaction) + ) + action = "created" + elif row.reaction == reaction: + # Toggle off + await session.execute( + _text(""" + DELETE FROM rss_item_reactions + WHERE user_id = :uid AND rss_item_id = :item_id + """).bindparams(uid=g.user.id, item_id=rss_item_id) + ) + action = "removed" + else: + # Flip + await session.execute( + _text(""" + UPDATE rss_item_reactions SET reaction = :reaction + WHERE user_id = :uid AND rss_item_id = :item_id + """).bindparams(reaction=reaction, uid=g.user.id, item_id=rss_item_id) + ) + action = "updated" + + await session.commit() + + return jsonify({"ok": True, "action": action}) + + +@briefing_bp.route("/rss-reactions/", methods=["DELETE"]) +@_REQUIRE +async def delete_rss_reaction(item_id: int): + """Explicitly remove a reaction (useful for MCP/external API callers).""" + from sqlalchemy import text as _text + + async with async_session() as session: + await session.execute( + _text(""" + DELETE FROM rss_item_reactions + WHERE user_id = :uid AND rss_item_id = :item_id + """).bindparams(uid=g.user.id, item_id=item_id) + ) + await session.commit() + + return jsonify({"ok": True}) +``` + +- [ ] **Step 2: Run the test suite** + +```bash +make test +``` + +Expected: all PASS. + +- [ ] **Step 3: Commit** + +```bash +git add src/fabledassistant/routes/briefing.py +git commit -m "feat(briefing): add POST/DELETE /api/briefing/rss-reactions endpoints" +``` + +--- + +## Task 11: Frontend API Helpers + +**Files:** +- Modify: `frontend/src/api/client.ts` + +- [ ] **Step 1: Add the two new API helpers to `client.ts`** + +Find the section at the end of `client.ts` where other helpers are defined and add: + +```typescript +export async function postRssReaction( + rssItemId: number, + reaction: 'up' | 'down' +): Promise<{ ok: boolean; action: string }> { + return apiPost('/api/briefing/rss-reactions', { rss_item_id: rssItemId, reaction }); +} + +export async function deleteRssReaction(rssItemId: number): Promise<{ ok: boolean }> { + return apiDelete(`/api/briefing/rss-reactions/${rssItemId}`); +} +``` + +- [ ] **Step 2: Run TypeScript check** + +```bash +make typecheck +``` + +Expected: no errors. + +- [ ] **Step 3: Commit** + +```bash +git add frontend/src/api/client.ts +git commit -m "feat(briefing): add postRssReaction and deleteRssReaction API helpers" +``` + +--- + +## Task 12: WeatherCard.vue + +**Files:** +- Create: `frontend/src/components/WeatherCard.vue` + +- [ ] **Step 1: Create the component** + +```vue + + + + + +``` + +- [ ] **Step 2: TypeScript check** + +```bash +make typecheck +``` + +Expected: no errors. + +- [ ] **Step 3: Commit** + +```bash +git add frontend/src/components/WeatherCard.vue +git commit -m "feat(briefing): add WeatherCard.vue component" +``` + +--- + +## Task 13: BriefingView.vue — Metadata Integration + +**Files:** +- Modify: `frontend/src/views/BriefingView.vue` + +- [ ] **Step 1: Read the current `BriefingView.vue` to understand its structure** + +```bash +docker compose exec app cat frontend/src/views/BriefingView.vue | head -100 +``` + +Or use the Read tool. Understand: how messages are loaded, how they are rendered, where to insert the WeatherCard. + +- [ ] **Step 2: Add WeatherCard import and type definitions** + +At the top of ` + +