docs: remove extraneous content — pipeline internals moved to architecture, changelog removed
- features.md: remove SQL impl detail from tasks section, sw.js reference from PWA section, and entire "LLM Chat — Internal Pipeline" section (moved to architecture.md) - architecture.md: add "LLM Pipeline Internals" section (intent routing, tool loop, duplicate guards, context window, research pipeline, image cache) - development.md: remove site-specific NFS path from custom runner instructions - Remove changelog.md (duplicates git history) - README.md: remove changelog link Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -267,6 +267,52 @@ Session cookies: `HttpOnly`, `SameSite=Lax`, optionally `Secure`. Session includ
|
||||
|
||||
See [sso-oauth.md](sso-oauth.md) for provider-specific setup instructions.
|
||||
|
||||
## LLM Pipeline Internals
|
||||
|
||||
### Intent Routing
|
||||
|
||||
Before the main model runs, a lightweight intent classifier (`services/intent.py`) runs concurrently with `build_context()`. It makes a fast non-streaming call using a smaller dedicated model (`OLLAMA_INTENT_MODEL`, default `qwen2.5:7b`) to determine if the message requires a tool call.
|
||||
|
||||
**Skip heuristic** — Intent classification is skipped entirely for short messages (≤10 words) with no action/object keywords, saving 400–800ms on conversational replies.
|
||||
|
||||
**Prior-work fast-path** — `_PRIOR_WORK_REFS` regex detects phrases like "research you did", "note you made", "using your research" and returns no-tool immediately, preventing `search_web` from firing when the user references existing notes.
|
||||
|
||||
If a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk (TTFT), the tool executes, then the main model generates a follow-up with the tool result. For chat-only responses the main model streams directly.
|
||||
|
||||
### Tool Loop
|
||||
|
||||
Multi-round tool loop (max 5 rounds). All implementations in `services/tools.py`; `execute_tool()` is the dispatcher.
|
||||
|
||||
**Duplicate protection on `create_note` / `create_task`:**
|
||||
1. Exact title match (case-insensitive) → hard block, redirect to `update_note`
|
||||
2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block
|
||||
3. Semantic content similarity (threshold 0.90, body ≥ 200 chars) → soft block with `requires_confirmation: true`
|
||||
|
||||
**Project resolution** (`_resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55.
|
||||
|
||||
### Context Window and Summarisation
|
||||
|
||||
`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure.
|
||||
|
||||
History summarisation threshold: 30 messages. Keeps 8 recent messages. Summary max 400 tokens.
|
||||
|
||||
### Web Research Pipeline
|
||||
|
||||
`services/research.py` implements a full autonomous research pipeline:
|
||||
1. Intent model generates 5 focused sub-queries
|
||||
2. All 5 SearXNG queries run in parallel (200ms stagger to avoid rate limiter)
|
||||
3. Up to 15 unique URLs fetched in parallel
|
||||
4. Up to 12 sources passed to synthesis LLM
|
||||
5. Result saved as a note with `tags=["research"]`
|
||||
|
||||
SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for trusted backend requests.
|
||||
|
||||
### Image Cache
|
||||
|
||||
`search_images` tool fetches images server-side via SearXNG, stores them on disk (SHA-256 dedup, content-type validation, 5 MB cap), and serves from `/api/images/<id>`. The user's browser never contacts the original image host.
|
||||
|
||||
Config: `IMAGE_CACHE_DIR` (default `/data/images`), `IMAGE_MAX_BYTES` (default 5 MB).
|
||||
|
||||
## RAG Pipeline
|
||||
|
||||
1. `semantic_search_notes()` — cosine similarity via pgvector, threshold configurable per call.
|
||||
|
||||
@@ -1,150 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
Development history — most recent first.
|
||||
|
||||
---
|
||||
|
||||
## 2026-03-24 — fable-mcp serve, admin log tool, Settings MCP install UI
|
||||
|
||||
- **Dockerfile** — builds `fable-mcp` wheel into `/app/dist/` at image build time
|
||||
- **`routes/fable_mcp_dist.py`** — `GET /api/fable-mcp/info` (availability check) and `GET /api/fable-mcp/download` (wheel download)
|
||||
- **`fable_mcp/tools/admin.py`** — `get_app_logs()` calling `/api/admin/logs`
|
||||
- **`fable_mcp/server.py`** — `fable_get_app_logs` MCP tool
|
||||
- **SettingsView** — "Fable MCP" section in API Keys tab: wheel download button and install instructions
|
||||
- **ci.yml** — `fable-mcp/**` added to trigger paths
|
||||
|
||||
## 2026-03-23 — fable-mcp scaffold + all tools
|
||||
|
||||
- `fable-mcp/` Python package scaffolded (`pyproject.toml`, `hatchling` build)
|
||||
- `FableClient` — async httpx wrapper with `FABLE_URL` + `FABLE_API_KEY` auth
|
||||
- Tool modules: `notes`, `tasks`, `projects`, `milestones`, `search`, `chat`, `admin`
|
||||
- `server.py` — FastMCP stdio transport; all tools registered
|
||||
- Settings UI — API Keys tab with create/revoke, `.env` download, Claude config download
|
||||
- `routes/api_keys.py` — `POST /api/api-keys`, `GET /api/api-keys`, `DELETE /api/api-keys/:id`
|
||||
|
||||
## 2026-03-23 — Model management UI, RSS category/refresh, weather city search
|
||||
|
||||
- **SettingsView** — Model Management section: list installed models (size, loaded state), pull by name (SSE progress), delete unused
|
||||
- **`routes/chat.py` `list_models_route`** — also hits `/api/ps` in parallel; returns `loaded: bool` and `modified_at` per model
|
||||
- **RSS feeds** — category badge, last-fetched age, manual refresh button, category input on add
|
||||
- **`routes/briefing.py`** — `POST /api/briefing/feeds/refresh` triggers immediate re-fetch
|
||||
- **`get_weather` tool** — now accepts any city name (geocodes via Nominatim + fetches Open-Meteo); not limited to pre-configured locations
|
||||
|
||||
## 2026-03-22 — Briefing scheduler fix, quick capture fix, task back navigation
|
||||
|
||||
- **Briefing scheduler** — fixed deadlock: `start_briefing_scheduler` made async; replaced `run_coroutine_threadsafe(...).result()` with `await` (was blocking the event loop thread)
|
||||
- **Briefing pipeline** — added early return when both LLM synthesis lanes produce empty strings (was always prepending "Good morning" on failure)
|
||||
- **Quick capture** — fixed: was always creating notes. Two causes fixed: (1) now uses user's `default_model` setting instead of `Config.OLLAMA_MODEL`; (2) removed `create_project` from classifier prompt (tool not offered, causing fallback to note)
|
||||
- **TaskViewerView** — back button navigates to task's project when `project_id` is set; Esc handler registered in capture phase to prevent conflict with App.vue global handler
|
||||
|
||||
## 2026-03-21 — Briefing RSS + project kanban advance buttons
|
||||
|
||||
- **ProjectView kanban** — status-advance buttons on task cards (→ todo→in_progress, ✓ in_progress→done), hidden until hover
|
||||
- **Briefing UI** — RSS feeds section: add URL + category, list feeds with last-fetched age badge, delete, manual refresh
|
||||
|
||||
## 2026-03-16 — Fuzzy project resolution, create_project guard, task embedding fix
|
||||
|
||||
- `services/tools.py` — `_resolve_project` rewritten with 4-step lookup: (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55. Partial names now reliably resolve without creating duplicates.
|
||||
- `services/tools.py` — `create_project` handler: similar-project checks (threshold 0.55) run **before** the `confirmed` gate, so `confirmed=true` cannot bypass them.
|
||||
- `routes/tasks.py` — fire-and-forget `upsert_note_embedding` added to create and update routes (tasks weren't getting embeddings via REST API).
|
||||
- `services/briefing_pipeline.py` — `await is_caldav_configured()` fix (was called without await — latent bug since the function became async).
|
||||
- `services/notes.py` — removed erroneous duplicate `count_query.where(Note.parent_id == parent_id)` that was applied inside the `no_project` branch.
|
||||
|
||||
## 2026-03-12 — CI release process update
|
||||
|
||||
- `.forgejo/workflows/ci.yml` — removed `main` from `branches` push trigger. CI fires only on `dev` pushes, `v*` tags. Production release: create release via Forgejo UI on `main` with a `v*` tag → tag push fires CI → build job pushes `:latest` + `:<version>`.
|
||||
|
||||
## 2026-03-11 — Daily Briefing (full feature)
|
||||
|
||||
Backend:
|
||||
- Migration `0026_add_briefing_tables.py`: `rss_feeds`, `rss_items`, `weather_cache`; added `conversation_type` and `briefing_date` to `conversations`.
|
||||
- `services/weather.py`: geocoding via Nominatim, forecast via Open-Meteo, per-user DB cache.
|
||||
- `services/rss.py`: feedparser-based fetch, per-feed DB cache prune-to-100.
|
||||
- `services/briefing_pipeline.py`: two-lane parallel gather (weather, RSS, tasks, calendar) → LLM synthesis → GenerationBuffer SSE stream.
|
||||
- `services/briefing_scheduler.py` (APScheduler): slots morning/midday/evening/night with catch-up logic.
|
||||
- `routes/briefing.py`: RSS CRUD at `/api/briefing/feeds`, weather config, briefing config, `POST /api/briefing/trigger`.
|
||||
|
||||
Frontend:
|
||||
- `BriefingView.vue`: primary briefing page; digest card + conversation thread + reply bar.
|
||||
- `BriefingSetupWizard.vue`: 4-step setup wizard shown on first visit.
|
||||
- SettingsView: Briefing tab (enable toggle, location geocoding, office days, time slot toggles, RSS CRUD).
|
||||
- AppHeader: "Briefing" nav link added.
|
||||
|
||||
## 2026-03-11 — Version footer, task history, note version retention
|
||||
|
||||
- CalVer build-time version tracking: `Dockerfile` injects `BUILD_VERSION` ARG; displayed in Settings footer.
|
||||
- Task history moved to collapsible right sidebar (previously modal overlay).
|
||||
- Note version retention: configurable max versions per note (default 20) enforced at write time.
|
||||
- Content deduplication in `build_context()` — same note cannot appear in multiple context blocks.
|
||||
|
||||
## 2026-03-11 — Multi-user sharing, notifications, groups, backup rewrite
|
||||
|
||||
- Migration `0025_add_sharing_and_notifications.py`: `groups`, `group_memberships`, `project_shares`, `note_shares`, `notifications`.
|
||||
- `services/access.py`: `PERMISSION_RANK` dict; `get_project_permission` / `get_note_permission` — check ownership, direct share, group-based share, note→project inheritance.
|
||||
- Share dialog (`ShareDialog.vue`): user tab (search + permission select), group tab, current shares list with inline change/remove.
|
||||
- `SharedWithMeView.vue`: `/shared` route — shared projects and notes.
|
||||
- In-app notification bell with 60s polling; push generated on share/group events.
|
||||
- Backup rewrite (version 2): includes all tables; `restore_full_backup` builds ID maps, two-pass note restore for `parent_id` patching.
|
||||
|
||||
## 2026-03-10 — Project Workspace, Graph View, settings tabs, message queue persistence
|
||||
|
||||
- `WorkspaceView.vue`: 3-panel layout (tasks/chat/notes), collapse toggles, SSE watcher auto-loads notes and refreshes tasks.
|
||||
- `WorkspaceTaskPanel.vue`: milestone-grouped task list, detail slide-over with status cycle + log.
|
||||
- `WorkspaceNoteEditor.vue`: list ↔ TipTap editor, autosave.
|
||||
- `GraphView.vue`: D3 force-directed graph, tag nodes, project hub nodes, physics panel, peek panel.
|
||||
- SettingsView fully rewritten with tabbed layout (General, Account, Notifications, Integrations, Data, Admin).
|
||||
- Message queue: persisted to localStorage; queued bubbles shown in chat with opacity + "QUEUED" label.
|
||||
- ToolCallCard: confirm/deny inline; "Create anyway" posts directly to API (no LLM round-trip).
|
||||
- Duplicate detection: scoped to same `is_task` type; semantic threshold raised 0.87 → 0.90.
|
||||
|
||||
## 2026-03-10 — "Illuminated Transcript" design language
|
||||
|
||||
- Fraunces font for headings; assistant bubbles elevated with left indigo accent + shadow; user bubbles ghosted (transparent bg, muted text).
|
||||
- Dark palette shifted to slate-indigo (`#111113`). `--radius-lg` 12→18px.
|
||||
- Cards: shadow depth replaces borders. Send buttons: indigo gradient + glow.
|
||||
|
||||
## 2026-03-10 — CI/CD infrastructure
|
||||
|
||||
- `infra/Dockerfile.runner-base`: Ubuntu 24.04 + Python 3.12 + Node 22 LTS custom runner image.
|
||||
- All CI jobs use `py3.12-node22` runner; pip caching added; Node bumped to 22.
|
||||
|
||||
## 2026-03-06 — Session invalidation on credential change
|
||||
|
||||
- `users.session_version` column; incremented on password change/reset; checked in `_check_auth`; mismatch evicts all other sessions.
|
||||
|
||||
## 2026-03-06 — DRY refactoring (backend + frontend)
|
||||
|
||||
- `models/base.py`: `TimestampMixin` + `CreatedAtMixin` applied to all models (~60 lines removed).
|
||||
- `routes/utils.py`: `not_found()` + `parse_iso_date()` helpers.
|
||||
- Frontend composables: `useAutoSave`, `useEditorGuards`, `useTagSuggestions`, `useFloatingAssist`, `useListKeyboardNavigation`.
|
||||
- Shared CSS: `editor-shared.css`, `viewer-shared.css`. Net: −634 lines, +237 lines.
|
||||
|
||||
## 2026-03-06 — Project-aware writing assist, link suggestions, project-scoped RAG
|
||||
|
||||
- Writing assist: `context_notes` param injects project notes; `definition`-tagged notes prioritised.
|
||||
- Link suggestions: detects note titles as plain text in body; one-click wikilink conversion.
|
||||
- Project-scoped RAG: `rag_project_id` param restricts semantic + keyword search to a project.
|
||||
- `search_notes` tool: upgraded to hybrid semantic + keyword; `relevance` field in results.
|
||||
- SSRF protection: `_is_private_url()` blocks private/loopback addresses; `follow_redirects=False`.
|
||||
- Config validation: `Config.validate()` at startup.
|
||||
|
||||
## 2026-03-05 — Note editor sidebar, full-document writing assist, drafts, version history
|
||||
|
||||
- NoteEditorView: two-column layout; writing assistant always visible in sidebar.
|
||||
- `whole_doc` mode: assistant outputs complete revised document (not just selected section).
|
||||
- `DiffView.vue`: standalone diff component with equal-line collapsing (git-style context).
|
||||
- Persistent drafts: `note_drafts` table; draft auto-saved after assist generation; restored on load.
|
||||
- `note_versions` table: auto-snapshot on body change; version browser with diff + restore.
|
||||
|
||||
## 2026-03-05 — Task work log, task editor overhaul
|
||||
|
||||
- `task_logs` table; `TaskLogSection.vue` with timestamps, duration badges, inline edit.
|
||||
- TaskEditorView: two-column layout with collapsible sidebar; sub-tasks section.
|
||||
- `log_work` LLM tool added to `_CORE_TOOLS`.
|
||||
|
||||
## 2026-03-04 — Per-conversation streaming, SSE reconnection, keyboard navigation
|
||||
|
||||
- Per-conversation streaming state; auto-new chat on open.
|
||||
- SSE reconnection: chat `reconnectIfGenerating()` on mount + navigation; assist retry loop with `Last-Event-ID`.
|
||||
- Model load timeout raised 90s → 180s.
|
||||
- Writing assist retry loop (3 attempts, exponential backoff).
|
||||
+1
-4
@@ -83,10 +83,7 @@ Runner base image: `infra/Dockerfile.runner-base` (Ubuntu 24.04 + Python 3.12 +
|
||||
Runner config: `infra/act-runner-config.yml` (label: `py3.12-node22`).
|
||||
Runner compose: `infra/runner-compose.yml`.
|
||||
|
||||
To activate a new runner registration:
|
||||
1. Copy `infra/act-runner-config.yml` → `/nfs/data/fabledsword/act_runner/config.yml`
|
||||
2. Delete `/data/.runner` in the runner container
|
||||
3. Restart the stack
|
||||
To activate a new runner registration, copy `infra/act-runner-config.yml` to the runner's config directory, delete the `.runner` registration file in the runner container, and restart the stack.
|
||||
|
||||
### Docker Registry
|
||||
|
||||
|
||||
+2
-53
@@ -16,7 +16,7 @@ Write in Markdown with a live-preview editor (Tiptap/ProseMirror). Headings, bol
|
||||
|
||||
## Tasks
|
||||
|
||||
Tasks are notes with `status IS NOT NULL`. They carry status (`todo` → `in_progress` → `done`), priority (`none`/`low`/`medium`/`high`), due date, milestone assignment, and a parent task (sub-tasks).
|
||||
Tasks carry status (`todo` → `in_progress` → `done`), priority (`none`/`low`/`medium`/`high`), due date, milestone assignment, and a parent task (sub-tasks).
|
||||
|
||||
**Task work logs** — Append progress log entries to a task with optional duration. Time tracking is visible in the task editor sidebar.
|
||||
|
||||
@@ -97,7 +97,7 @@ Quick capture from the Android app routes to the intent classifier. It creates n
|
||||
|
||||
## PWA
|
||||
|
||||
Installable as a desktop or mobile app. Service worker caches the shell. Push notifications handled by `public/sw.js` (suppresses duplicate notifications when the relevant tab is already focused). Works over HTTPS only in Firefox.
|
||||
Installable as a desktop or mobile app. Service worker caches the shell; push notifications are suppressed when the relevant tab is already focused. Works over HTTPS only in Firefox.
|
||||
|
||||
## Settings
|
||||
|
||||
@@ -117,57 +117,6 @@ Settings are tabbed:
|
||||
| Logs (admin) | Error, audit, and usage logs with search |
|
||||
| Groups (admin) | Create/manage groups and membership |
|
||||
|
||||
## LLM Chat — Internal Pipeline
|
||||
|
||||
### Intent Routing
|
||||
|
||||
Before the main model runs, a lightweight intent classifier (`services/intent.py`) runs concurrently with `build_context()`. It makes a fast non-streaming call (~400ms) using a smaller dedicated model (`OLLAMA_INTENT_MODEL`, default `qwen2.5:7b`) to determine if the message requires a tool call.
|
||||
|
||||
**Skip heuristic** — Intent classification is skipped entirely for short messages (≤10 words) with no action/object keywords, saving 400–800ms on conversational replies.
|
||||
|
||||
**Prior-work fast-path** — Phrases like "research you did", "note you made", "using your research" skip the LLM call entirely and route to null (chat), preventing `search_web` from firing when the user references existing notes.
|
||||
|
||||
If a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk (becoming time-to-first-token), then the tool executes, then the main model generates a follow-up response with the tool result.
|
||||
|
||||
### Tool Loop
|
||||
|
||||
Multi-round tool loop (max 5 rounds). All tool implementations are in `services/tools.py` with `execute_tool()` as the dispatcher.
|
||||
|
||||
**Duplicate protection on create_note / create_task:**
|
||||
1. Exact title match (case-insensitive) → hard block, redirect to `update_note`
|
||||
2. Fuzzy title match (SequenceMatcher ≥ 82%; punctuation stripped before candidate search) → hard block
|
||||
3. Semantic content similarity (`semantic_search_notes` threshold 0.90, body ≥ 200 chars) → soft block with `requires_confirmation: true`
|
||||
|
||||
**Project resolution** (`_resolve_project`): 4-step lookup — (1) exact DB match, (2) `query in title` substring, (3) `title in query` reverse substring, (4) SequenceMatcher ≥ 0.55.
|
||||
|
||||
### Model Status
|
||||
|
||||
Three states returned by `GET /api/chat/status`:
|
||||
- `not_found` — model not installed; orange indicator in header
|
||||
- `cold` — installed but not loaded in VRAM; yellow pulsing indicator (first request will be slow)
|
||||
- `loaded` — hot in VRAM; green indicator
|
||||
|
||||
After `warmModel()` is called, a 5s polling loop runs for up to 60s until the indicator turns green.
|
||||
|
||||
### Context Window
|
||||
|
||||
`OLLAMA_NUM_CTX` (default 16384) controls the context window for all generation calls. Intent classification always uses `num_ctx=4096` to reduce VRAM pressure. History summarisation kicks in at 30 messages, keeps 8 recent, summary max 400 tokens.
|
||||
|
||||
### Image Search
|
||||
|
||||
`search_images` tool (SearXNG `categories=images`) fetches images server-side, stores on disk (SHA-256 dedup, 5 MB cap), and serves from `/api/images/<id>` — the user's browser never contacts the original image host. Requires `SEARXNG_URL`.
|
||||
|
||||
### Web Research Pipeline
|
||||
|
||||
Triggered by "research X and make a note" or the Research button in ChatView:
|
||||
1. Intent model generates 5 focused sub-queries as a JSON array
|
||||
2. All 5 SearXNG queries run in parallel (200ms stagger)
|
||||
3. Up to 15 unique URLs fetched in parallel
|
||||
4. Up to 12 sources passed to synthesis LLM (`num_ctx=16384`, `num_predict=8192`)
|
||||
5. Note created with `tags=["research"]`
|
||||
|
||||
SearXNG tip: add the app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml` to bypass the rate limiter for backend requests.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- Calendar/timeline view for tasks with due dates
|
||||
|
||||
Reference in New Issue
Block a user