diff --git a/README.md b/README.md index c0615d2..f28cb98 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ Go to **Settings → General** to pull an LLM model. `qwen3:8b` or `llama3.1:8b` | [Development](docs/development.md) | Dev workflow, CI/CD, migrations, release process | | [API Keys & MCP](docs/api-keys-and-mcp.md) | API key management and Fable MCP install guide | | [SSO / OAuth](docs/sso-oauth.md) | OIDC setup for Authentik, Keycloak, and other providers | +| [API Reference](docs/api-reference.md) | All REST API endpoints | +| [Android App](docs/android-app.md) | Flutter companion app architecture and feature status | | [Changelog](docs/changelog.md) | Development history | ## License diff --git a/docs/android-app.md b/docs/android-app.md new file mode 100644 index 0000000..ef9c0ee --- /dev/null +++ b/docs/android-app.md @@ -0,0 +1,73 @@ +# Android Companion App + +The Android companion app lives in a separate repository at `/home/bvandeusen/Nextcloud/Projects/fabled_app`. + +## Stack + +- Flutter + Dart +- Riverpod (state management) +- GoRouter (navigation) +- Dio (HTTP client) +- PersistCookieJar (session persistence) +- SSE streaming via `fetch` + `ReadableStream` bridge + +## Architecture + +``` +lib/ + app.dart # GoRouter + _Shell + _QuickCaptureBar + core/constants.dart # Routes.* + data/ + models/ # note.dart, task.dart, project.dart + api/ # notes_api.dart, tasks_api.dart, projects_api.dart + repositories/ # notes, tasks, projects repositories + providers/ + api_client_provider.dart # all API + repository providers + notes_provider.dart # NotesNotifier + tasks_provider.dart # TasksNotifier + projects_provider.dart # ProjectsNotifier + screens/ + notes/note_edit_screen.dart # chip tag input + ProjectSelector + tasks/task_edit_screen.dart # ProjectSelector + projects/project_list_screen.dart + widgets/ + project_selector.dart # reusable DropdownButtonFormField +``` + +## Navigation + +4-tab shell (Notes · Tasks · Projects · Chat): +- Phone: bottom `NavigationBar` +- Tablet/landscape: `NavigationRail` + +Quick Capture bar persists across all tabs. Settings accessible from top-right icon. + +## Feature Status + +| Feature | Status | Notes | +|---------|--------|-------| +| Notes CRUD | ✅ | Tags chip input; project selector in editor | +| Tasks CRUD | ✅ | Project selector in editor | +| Projects list | ✅ | Active/archived sections; long-press status change; create dialog | +| Chat + SSE | ✅ | Full streaming | +| Quick Capture | ✅ | Offline queue with retry | +| Tags | ✅ | Chip input in NoteEditScreen; typed as `List` | +| Project assignment | ✅ | `ProjectSelector` dropdown in Note + Task editors | +| Milestones | ❌ deferred | Too granular for mobile; web UI handles it | +| Push notifications | ❌ incompatible | Backend uses browser VAPID; Flutter needs FCM/APNs — separate implementation required | +| CalDAV settings | ❌ intentional | Server-side config only; not exposed in mobile app | + +## API Compatibility Notes + +- `GET /api/projects/:id` returns a flat JSON object (not `{project: ...}` wrapper); includes `summary` field. +- `POST /api/projects` returns the project dict directly (201). +- `PATCH /api/projects/:id` returns the updated project dict. +- Task body field is `body` (not `description`) — the app maps `description` → `body` on serialize. + +## Self-Update + +The app supports self-update via the Forgejo release API (`update_provider.dart`). It checks the latest release tag and prompts the user to download and install a new APK when one is available. + +## CI + +Builds are triggered from the Forgejo Actions pipeline in the `fabled_app` repository. The APK is attached to the release as a downloadable artifact. diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 0000000..1a571ca --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,244 @@ +# API Reference + +All endpoints require login (session cookie or `Authorization: Bearer `) unless marked **(public)**. + +## Health + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/health` | Health check **(public)** | + +## Auth + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/auth/status` | `{has_users, registration_open, oauth_enabled, local_auth_enabled}` **(public)** | +| POST | `/api/auth/register` | Register new user (first user becomes admin; 403 if registration closed or local auth disabled) | +| POST | `/api/auth/login` | Login with username/password (403 if local auth disabled) | +| POST | `/api/auth/logout` | Clear session | +| GET | `/api/auth/me` | Current user info (includes `has_password: bool`) | +| PUT | `/api/auth/password` | Change password `{current_password, new_password}` | +| PUT | `/api/auth/email` | Change email `{email, password?}` (password required only for local-auth users) | +| POST | `/api/auth/invalidate-sessions` | Bump `session_version` — evicts all other sessions, keeps current alive | +| POST | `/api/auth/forgot-password` | Send password reset email `{email}` | +| POST | `/api/auth/reset-password` | Reset password with token `{token, new_password}` | +| GET | `/api/auth/oauth/login` | Initiate OIDC PKCE flow → redirect to provider | +| GET | `/api/auth/oauth/callback` | OIDC callback — exchange code, find/create user, redirect to `/` | +| GET | `/api/auth/invitation/:token` | Validate invitation token **(public)** | +| POST | `/api/auth/register-with-invite` | Register with token `{token, username, password}` **(public)** | + +## Notes + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/notes` | List notes. Params: `q`, `tag`, `sort`, `order`, `limit`, `offset`, `project_id`, `milestone_id`, `parent_id`, `type` (`note`/`task`/`all`) | +| POST | `/api/notes` | Create note `{title, body, tags?, status?, priority?, due_date?, project_id?, milestone_id?, parent_id?}` | +| GET | `/api/notes/tags` | All tags (param: `q` for filter) | +| POST | `/api/notes/suggest-tags` | LLM tag suggestions `{title, body, current_tags?}` → `{suggested_tags}` | +| POST | `/api/notes/link-suggestions` | Detect note titles as plain text in body `{body, project_id, exclude_note_id}` → `[{note_id, title, count}]` | +| GET | `/api/notes/by-title` | Resolve note by exact title (param: `title`) | +| POST | `/api/notes/resolve-title` | Get-or-create note by title `{title}` (wikilink click) | +| GET | `/api/notes/:id` | Get single note | +| PUT | `/api/notes/:id` | Full update | +| PATCH | `/api/notes/:id` | Partial update (same fields as PUT) | +| DELETE | `/api/notes/:id` | Delete note | +| POST | `/api/notes/:id/convert-to-task` | Set `status='todo'`, `priority='none'` | +| POST | `/api/notes/:id/convert-to-note` | Clear `status`, `priority`, `due_date` | +| POST | `/api/notes/:id/append-tag` | Add tag `{tag}` → updated note | +| GET | `/api/notes/:id/backlinks` | Notes/tasks with `[[Title]]` references to this note | +| GET | `/api/notes/:id/versions` | List note version history | +| GET | `/api/notes/:id/versions/:vid` | Get a specific version | +| GET | `/api/notes/:id/draft` | Get current AI draft | +| PUT | `/api/notes/:id/draft` | Save AI draft | +| DELETE | `/api/notes/:id/draft` | Delete AI draft | +| POST | `/api/notes/assist` | Launch AI assist generation → 202 `{body, target_section?, instruction, whole_doc?}` | +| GET | `/api/notes/assist/stream` | SSE stream for assist (Last-Event-ID reconnect; events: `chunk`, `done`, `error`) | + +## Tasks + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/tasks` | List tasks. Params: `q`, `tag`, `status`, `priority`, `due_before`, `due_after`, `sort`, `order`, `limit`, `offset` | +| POST | `/api/tasks` | Create task (accepts `project` name string → resolved to `project_id`) | +| GET | `/api/tasks/:id` | Get task (includes `parent_title`) | +| PUT | `/api/tasks/:id` | Full update | +| PATCH | `/api/tasks/:id/status` | Quick status update `{status}` | +| DELETE | `/api/tasks/:id` | Delete task | +| GET | `/api/tasks/:id/logs` | List work logs | +| POST | `/api/tasks/:id/logs` | Create log `{content, duration_minutes?}` | +| PATCH | `/api/tasks/:id/logs/:log_id` | Update log | +| DELETE | `/api/tasks/:id/logs/:log_id` | Delete log | + +## Projects & Milestones + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/projects` | List projects (owned + shared) | +| POST | `/api/projects` | Create project | +| GET | `/api/projects/:id` | Get project with `milestone_summary` | +| PATCH | `/api/projects/:id` | Update project | +| DELETE | `/api/projects/:id` | Delete project | +| GET | `/api/projects/:id/notes` | Notes + tasks in this project | +| GET | `/api/projects/:id/milestones` | List milestones | +| POST | `/api/projects/:id/milestones` | Create milestone | +| PATCH | `/api/projects/:id/milestones/:mid` | Update milestone | +| DELETE | `/api/projects/:id/milestones/:mid` | Delete milestone | + +## Sharing + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/projects/:id/shares` | List project shares | +| POST | `/api/projects/:id/shares` | Create project share `{user_id?, group_id?, permission}` | +| PATCH | `/api/projects/:id/shares/:sid` | Update permission | +| DELETE | `/api/projects/:id/shares/:sid` | Remove share | +| GET | `/api/notes/:id/shares` | List note shares | +| POST | `/api/notes/:id/shares` | Create note share | +| PATCH | `/api/notes/:id/shares/:sid` | Update permission | +| DELETE | `/api/notes/:id/shares/:sid` | Remove share | +| GET | `/api/shared-with-me` | All resources shared with the current user | + +## Groups + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/groups` | List all groups (admin only) | +| POST | `/api/groups` | Create group `{name, description?}` | +| PATCH | `/api/groups/:id` | Update group | +| DELETE | `/api/groups/:id` | Delete group | +| GET | `/api/groups/:id/members` | List members | +| POST | `/api/groups/:id/members` | Add member `{user_id, role}` | +| PATCH | `/api/groups/:id/members/:uid` | Update member role | +| DELETE | `/api/groups/:id/members/:uid` | Remove member | + +## Chat + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/chat/conversations` | List conversations (params: `limit`, `offset`) | +| POST | `/api/chat/conversations` | Create conversation `{title?, model?}` | +| POST | `/api/chat/conversations/bulk-delete` | Delete multiple conversations `{ids: number[]}` | +| GET | `/api/chat/conversations/:id` | Get conversation with all messages | +| PATCH | `/api/chat/conversations/:id` | Update title or model | +| DELETE | `/api/chat/conversations/:id` | Delete conversation (cascades to messages) | +| POST | `/api/chat/conversations/:id/messages` | Start generation → 202. Body: `{content, context_note_id?, include_note_ids?, rag_project_id?, workspace_project_id?, think?}` | +| GET | `/api/chat/conversations/:id/generation/stream` | SSE stream (Last-Event-ID reconnect; events: `context`, `chunk`, `tool_call`, `status`, `done`, `error`) | +| POST | `/api/chat/conversations/:id/generation/cancel` | Cancel active generation | +| POST | `/api/chat/messages/:id/save-as-note` | Save assistant message as note | +| POST | `/api/chat/conversations/:id/summarize` | Summarize conversation → note | +| GET | `/api/chat/status` | Ollama availability + model state `{ollama, model, default_model}` | +| GET | `/api/chat/models` | List installed Ollama models (includes `loaded: bool`, `modified_at`) | +| POST | `/api/chat/models/pull` | Pull model (SSE NDJSON progress) `{model}` | +| POST | `/api/chat/models/delete` | Delete model `{model}` | +| GET | `/api/chat/ps` | Currently loaded (hot) models | +| POST | `/api/chat/warm` | Pre-load model into VRAM `{model}` → 202 | + +## Quick Capture + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/quick-capture` | Classify + create item from natural language `{text}` → `{success, type, message, data}` | + +## Search + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/search` | Semantic + keyword search across notes and tasks. Params: `q`, `type` (`note`/`task`/`all`), `limit` | + +## Briefing + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/briefing/config` | Get briefing configuration | +| PUT | `/api/briefing/config` | Save briefing configuration | +| GET | `/api/briefing/feeds` | List RSS feeds | +| POST | `/api/briefing/feeds` | Add RSS feed `{url, name?, category?}` | +| DELETE | `/api/briefing/feeds/:id` | Delete feed | +| POST | `/api/briefing/feeds/refresh` | Trigger immediate feed refresh → `{feeds_refreshed, new_items}` | +| GET | `/api/briefing/weather` | Get weather configuration | +| PUT | `/api/briefing/weather` | Save weather locations | +| POST | `/api/briefing/weather/geocode` | Geocode address `{query}` → `{lat, lon, label}` | +| POST | `/api/briefing/trigger` | Manually fire a briefing slot `{slot}` | +| GET | `/api/briefing/conversations` | List past briefing conversations | +| GET | `/api/briefing/conversations/today` | Get/create today's briefing conversation | +| GET | `/api/briefing/conversations/:id/messages` | Get messages for a briefing conversation | + +## Settings + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/settings` | All settings as `{key: value}` | +| PUT | `/api/settings` | Update settings `{key: value, ...}` | +| GET | `/api/settings/models` | Installed models + defaults | +| GET | `/api/settings/search` | Proxy SearXNG search (params: `q`) | + +## API Keys + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/api-keys` | List user's API keys | +| POST | `/api/api-keys` | Create key `{name, scope}` → `{key, ...}` (key shown once) | +| DELETE | `/api/api-keys/:id` | Revoke key | + +## Fable MCP Distribution + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/fable-mcp/info` | `{available: bool, filename: string\|null}` | +| GET | `/api/fable-mcp/download` | Download wheel file | + +## Notifications + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/notifications` | List notifications | +| GET | `/api/notifications/count` | Unread count | +| POST | `/api/notifications/:id/read` | Mark read | +| POST | `/api/notifications/read-all` | Mark all read | + +## Push + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/push/vapid-public-key` | VAPID public key for subscription | +| POST | `/api/push/subscribe` | Register push subscription | +| DELETE | `/api/push/subscribe` | Unregister push subscription | + +## Images + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/images/:id` | Serve cached image **(no auth required — IDs are opaque SHA-256)** | + +## Users + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/users/search` | Search users by username/email prefix (param: `q`, min 2 chars, excludes self) | + +## Export + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/export` | Export data. Params: `format=markdown` (ZIP with `.md` + YAML frontmatter) or `format=json` | + +## Admin + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/admin/backup` | Export backup (`?scope=user` for own data; full requires admin) | +| POST | `/api/admin/restore` | Restore from JSON backup | +| GET | `/api/admin/users` | List all users | +| DELETE | `/api/admin/users/:id` | Delete user (cannot delete self) | +| GET | `/api/admin/registration` | Get registration open/closed state | +| PUT | `/api/admin/registration` | Toggle registration `{open: bool}` | +| POST | `/api/admin/invitations` | Create invitation `{email}` → sends email | +| GET | `/api/admin/invitations` | List pending invitations | +| DELETE | `/api/admin/invitations/:id` | Revoke invitation | +| GET | `/api/admin/logs` | Log entries. Params: `category`, `user_id`, `search`, `date_from`, `date_to`, `limit`, `offset` | +| GET | `/api/admin/logs/stats` | Log category counts | +| GET | `/api/admin/base-url` | Get base URL setting | +| PUT | `/api/admin/base-url` | Set base URL `{base_url}` | +| GET | `/api/admin/smtp` | Get SMTP config (password masked) | +| PUT | `/api/admin/smtp` | Save SMTP config | +| POST | `/api/admin/smtp/test` | Send test email `{recipient}` | diff --git a/docs/architecture.md b/docs/architecture.md index acc7596..02fe33d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -165,6 +165,85 @@ Permission resolution is centralised in `services/access.py`. `get_project_permi `category` (`audit`/`usage`/`error`), `user_id` FK nullable (SET NULL on delete), `username` (denormalised), `action`, `endpoint`, `method`, `status_code`, `duration_ms`, `error_type`, `error_message`, `traceback`, `details` JSONB. +## Detailed File Reference + +### Backend (`src/fabledassistant/`) + +| File | Responsibility | +|------|---------------| +| `app.py` | Quart app factory; SPA via 404 handler; JSON 404/500 for API; request logging; security headers in `after_request` | +| `auth.py` | `login_required`, `admin_required`, `get_current_user_id` — shared `_check_auth()` helper | +| `config.py` | All config from env vars + Docker secret file support (`_read_secret`); `SECURE_COOKIES`, `TRUST_PROXY_HEADERS`, `OLLAMA_NUM_CTX`; `oidc_enabled()`, `searxng_enabled()` classmethods | +| `rate_limit.py` | In-memory sliding-window rate limiter (`asyncio.Lock` + `defaultdict`); `is_rate_limited(key, max, window)` | +| `models/note.py` | Unified Note model (notes + tasks) | +| `models/user.py` | `id`, `username`, `email`, `password_hash` (nullable — NULL for OAuth-only), `oauth_sub` (unique nullable), `role`, `session_version` | +| `models/conversation.py` | `Conversation` + `Message` models | +| `models/app_log.py` | `AppLog` with `category`, denormalised `username`, `details` JSONB | +| `routes/api.py` | `/api` blueprint; `GET /api/health` (public) | +| `routes/auth.py` | Register, login, logout, me, password/email change, password reset, invite registration, OAuth login+callback; rate limiting; `LOCAL_AUTH_ENABLED` guards | +| `routes/admin.py` | Backup, restore, user management, registration toggle, invitations, base URL, SMTP (admin only) | +| `routes/chat.py` | Conversations CRUD; SSE generation stream; model pull/delete/list/warm; briefing conversation routes | +| `routes/notes.py` | Notes CRUD + wikilinks + backlinks + assist + link suggestions + version history + drafts | +| `routes/tasks.py` | Tasks CRUD; `POST` accepts `project` name string (resolved to `project_id`) | +| `routes/quick_capture.py` | `POST /api/quick-capture` — single-shot natural language item creation | +| `routes/search.py` | `GET /api/search` — semantic + keyword hybrid search | +| `services/auth.py` | `create_user`, `authenticate`, user lookups, password reset tokens, invitation tokens | +| `services/oauth.py` | OIDC discovery (cached), PKCE auth URL, code exchange, `find_or_create_oauth_user` | +| `services/llm.py` | `build_context()`, RAG injection, history summarisation, `stream_chat_with_tools()`, URL fetching, SSRF guard | +| `services/generation_task.py` | `run_generation()` — full chat pipeline: intent routing, tool loop, SSE fan-out, push notification; `run_assist_generation()` | +| `services/intent.py` | `classify_intent()` — fast non-streaming LLM call; intent skip heuristic; `_PRIOR_WORK_REFS` fast-path | +| `services/tools.py` | All LLM tool definitions + `execute_tool()` dispatcher; duplicate guards; `_resolve_project()` 4-step lookup | +| `services/embeddings.py` | `upsert_note_embedding()`, `semantic_search_notes()` (pgvector cosine similarity) | +| `services/generation_buffer.py` | In-memory SSE event buffer; `cancel_event`; 60s cleanup; supports both chat (int keys) and assist (string keys) | +| `services/briefing_pipeline.py` | Two-lane parallel gather → LLM synthesis → `GenerationBuffer` stream | +| `services/briefing_scheduler.py` | APScheduler `BackgroundScheduler`; slots with catch-up logic; async-safe via `asyncio.create_task` | +| `services/research.py` | SearXNG research pipeline: 5 sub-queries → parallel fetch → synthesis; `search_images` for image category | +| `services/caldav.py` | Full CalDAV event/todo lifecycle; synchronous library calls in asyncio executor | +| `services/images.py` | `fetch_and_store_image()` — SHA-256 dedup, content-type validation, 5 MB cap | +| `services/backup.py` | `export_full_backup()`, `export_user_backup()`, `restore_full_backup()` (version 2 with ID maps) | +| `services/push.py` | VAPID key auto-generation; `send_push_notification()` fire-and-forget; 410 Gone cleanup | +| `services/logging.py` | `log_audit`, `log_usage`, `log_error`, `start_log_retention_loop` (hourly cleanup) | +| `services/weather.py` | Nominatim geocoding + Open-Meteo forecast + per-user DB cache | +| `services/rss.py` | feedparser fetch; per-feed DB cache; prune-to-100 items | +| `services/assist.py` | `build_assist_messages()` — section/whole-doc modes; project context injection | + +### Frontend (`frontend/src/`) + +| File | Responsibility | +|------|---------------| +| `App.vue` | App shell (`100dvh` flex column); global keyboard shortcuts (`g`+key nav, `n/t/c/e///?`); starts status polling + loads settings on mount | +| `api/client.ts` | `ApiError`, `apiGet/Post/Put/Patch/Delete`, `apiSSEStream` (fetch + ReadableStream), auto 401→login redirect | +| `stores/auth.ts` | User, `isAuthenticated`, `isAdmin`, `oauthEnabled`, `localAuthEnabled`, login/register/logout/checkAuth | +| `stores/chat.ts` | Conversation CRUD; `sendMessage()` SSE streaming; `reconnectIfGenerating()`; message queue (localStorage); `streamingStatus` | +| `stores/notes.ts` | CRUD + tag filter; `resolveTitle`; `convertToTask/Note`; `fetchBacklinks`; `fetchAllTags` | +| `stores/tasks.ts` | CRUD + status/priority filter; `patchStatus` | +| `stores/settings.ts` | `assistantName`, `defaultModel`, `installedModels`; `pullModel()`, `deleteModel()` | +| `stores/push.ts` | `isSupported`, `permission`, `isSubscribed`, `subscribe/unsubscribe` | +| `stores/notifications.ts` | `count`, `items`, `fetchCount`, `fetchAll`, `markRead`, `markAll` | +| `views/HomeView.vue` | Chat-first dashboard; 6 task sections (Overdue → Other); 8 recent notes; inline streaming response | +| `views/ChatView.vue` | `/chat` — SSE streaming; context sidebar (Suggested/In Context); note picker; model selector; message queue; bulk delete | +| `views/WorkspaceView.vue` | `/workspace/:id` — 3-panel (tasks/chat/notes); SSE tool-call watcher; conv persisted to localStorage | +| `views/GraphView.vue` | `/graph` — D3 force-directed; tag/note/project-hub nodes; physics panel; peek panel | +| `views/ProjectView.vue` | Kanban grouped by milestone; advance buttons; milestone management | +| `views/SettingsView.vue` | Tabbed settings (11 tabs); tab state in localStorage | +| `views/NoteEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; diff view; version history; autosave | +| `views/TaskEditorView.vue` | Tiptap editor; 2-column layout; AI assist panel; task log section; sub-tasks | +| `components/ToolCallCard.vue` | Tool call results; `requires_confirmation` inline confirm/deny; direct POST on "Create anyway" | +| `components/ChatMessage.vue` | Message bubble; markdown rendering; tool call cards; "Save as Note" | +| `components/TagInput.vue` | Chip-based tag input; Enter/comma to confirm; autocomplete from `/api/notes/tags` | +| `components/TiptapEditor.vue` | Tiptap wrapper; markdown↔HTML round-trip; selection change emit; WikilinkDecoration; TagDecoration | +| `components/ShareDialog.vue` | `` modal; user tab + group tab; current shares list | +| `components/WorkspaceTaskPanel.vue` | Milestone-grouped task list; detail slide-over | +| `components/WorkspaceNoteEditor.vue` | List ↔ TipTap editor with autosave | +| `composables/useAssist.ts` | AI assist: section parsing, SSE streaming, accept/reject, LCS diff, persistent draft | +| `composables/useAutoSave.ts` | Interval-based autosave (5 min) with dirty/saving guards | +| `composables/useEditorGuards.ts` | Ctrl+S, `beforeunload` warning, route-leave confirm | +| `composables/useTagSuggestions.ts` | `/api/notes/suggest-tags` call + suggestion state | +| `composables/useListKeyboardNavigation.ts` | j/k/Enter list navigation; focus-in-input guard | +| `extensions/WikilinkDecoration.ts` | ProseMirror decoration plugin highlighting `[[wikilinks]]` | +| `extensions/TagDecoration.ts` | ProseMirror decoration plugin highlighting `#tags` | +| `extensions/WikilinkSuggestion.ts` | `@tiptap/suggestion` extension for `[[` autocomplete | + ## Key Services | Service | Responsibility | diff --git a/docs/development.md b/docs/development.md index febde61..3933534 100644 --- a/docs/development.md +++ b/docs/development.md @@ -97,6 +97,41 @@ Required secrets (repo → Settings → Secrets → Actions): - `REGISTRY_USER` — Forgejo username - `REGISTRY_TOKEN` — Forgejo PAT with `write:packages` scope +## Migration Chain + +Current migration sequence (all idempotent raw SQL): + +``` +0001 create_notes_table +0002 create_tasks_table +0003 task_note_companion (data migration) +0004 merge_tasks_into_notes +0005 add_chat_tables +0006 add_settings_table +0007 add_title_and_updated_at_indexes +0008 add_users_and_user_id +0009 add_message_status +0010 add_app_logs_table +0011 add_password_reset_tokens +0012 add_invitation_tokens +0013 add_tool_calls_to_messages +0014 add_note_embeddings +0015 add_oauth_fields +0016 add_image_cache +0017 add_projects +0018 add_push_subscriptions +0019 add_note_parent_and_project_fields +0020 add_milestones +0021 add_task_logs +0022 add_note_versions_and_drafts +0023 add_tags_to_note_versions +0024 add_session_version +0025 add_sharing_and_notifications +0026 add_briefing_tables +``` + +**Important:** Do NOT use `op.create_table()` or `sa.Enum()` — SQLAlchemy's event system can fire `CREATE TYPE` even with `create_type=False`, causing failures on re-run. Always use raw SQL with `IF NOT EXISTS` / `DO $$ BEGIN ... EXCEPTION WHEN duplicate_object` guards. + ## Project Conventions ### Backend diff --git a/docs/features.md b/docs/features.md index fcf1b55..0c87ad8 100644 --- a/docs/features.md +++ b/docs/features.md @@ -117,6 +117,65 @@ Settings are tabbed: | Logs (admin) | Error, audit, and usage logs with search | | Groups (admin) | Create/manage groups and membership | +## LLM Chat — Internal Pipeline + +### Intent Routing + +Before the main model runs, a lightweight intent classifier (`services/intent.py`) runs concurrently with `build_context()`. It makes a fast non-streaming call (~400ms) using a smaller dedicated model (`OLLAMA_INTENT_MODEL`, default `qwen2.5:7b`) to determine if the message requires a tool call. + +**Skip heuristic** — Intent classification is skipped entirely for short messages (≤10 words) with no action/object keywords, saving 400–800ms on conversational replies. + +**Prior-work fast-path** — Phrases like "research you did", "note you made", "using your research" skip the LLM call entirely and route to null (chat), preventing `search_web` from firing when the user references existing notes. + +If a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk (becoming time-to-first-token), then the tool executes, then the main model generates a follow-up response with the tool result. + +### Tool Loop + +Multi-round tool loop (max 5 rounds). All tool implementations are in `services/tools.py` with `execute_tool()` as the dispatcher. + +**Duplicate protection on create_note / create_task:** +1. Exact title match (case-insensitive) → hard block, redirect to `update_note` +2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block +3. Semantic content similarity (`semantic_search_notes` threshold 0.90, body ≥ 200 chars) → soft block with `requires_confirmation: true` + +**Project resolution** (`_resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55. + +### Model Status + +Three states returned by `GET /api/chat/status`: +- `not_found` — model not installed; orange indicator in header +- `cold` — installed but not loaded in VRAM; yellow pulsing indicator (first request will be slow) +- `loaded` — hot in VRAM; green indicator + +After `warmModel()` is called, a 5s polling loop runs for up to 60s until the indicator turns green. + +### Context Window + +`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure. History summarisation kicks in at 30 messages, keeps 8 recent, summary max 400 tokens. + +### Image Search + +`search_images` tool (SearXNG `categories=images`) fetches images server-side, stores on disk (SHA-256 dedup, 5 MB cap), and serves from `/api/images/` — the user's browser never contacts the original image host. Requires `SEARXNG_URL`. + +### Web Research Pipeline + +Triggered by "research X and make a note" or the Research button in ChatView: +1. Intent model generates 5 focused sub-queries as a JSON array +2. All 5 SearXNG queries run in parallel (200ms stagger) +3. Up to 15 unique URLs fetched in parallel +4. Up to 12 sources passed to synthesis LLM (`num_ctx=16384`, `num_predict=8192`) +5. Note created with `tags=["research"]` + +SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for backend requests. + +## Roadmap + +- Calendar/timeline view for tasks with due dates +- Email integration (read/send via IMAP/SMTP tools in chat) +- Session invalidation on user deletion +- Flutter push notifications (requires FCM/APNs — separate from web VAPID) +- Flutter milestone support in project view + ## Keyboard Shortcuts | Key | Action |