docs: remove dated design plans and one install's runner config
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m8s
CI & Build / Python tests (push) Successful in 1m55s
CI & Build / Build & push image (push) Successful in 39s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m8s
CI & Build / Python tests (push) Successful in 1m55s
CI & Build / Build & push image (push) Successful in 39s
At the operator's request (milestone 544, step 6203):
- docs/2026-03-23-fable-mcp-{design,plan}.md, 2026-03-29-voice-s2s-design.md
and 2026-04-10-agentic-briefing-design.md were plans for work that has
since shipped or been retired (voice and briefings left in the pivot).
Plans live in Scribe milestones.
- infra/act-runner-config.yml and runner-compose.yml described one
install's CI runner (an /nfs path, the py3.12-node22 label, a
RUNNER_SETUP.md that no longer exists). The runner is the CI-Runner
repo's concern.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,271 +0,0 @@
|
|||||||
# Fable MCP — Design Spec
|
|
||||||
|
|
||||||
**Date:** 2026-03-23
|
|
||||||
**Status:** Approved
|
|
||||||
**Author:** bvandeusen + Claude
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
A Python MCP (Model Context Protocol) server that lets Claude directly interface with a running Fabled Assistant instance. The goal is two-fold: Claude's stronger reasoning handles planning, documentation, and analysis while Fable remains the authoritative store for notes, tasks, projects, and milestones; and direct API access enables process improvement by letting Claude observe and operate on real data rather than working from descriptions.
|
|
||||||
|
|
||||||
The work is split into two sub-projects:
|
|
||||||
|
|
||||||
1. **Fable API Key Feature** — additions to the main `scribe` project to support bearer token authentication
|
|
||||||
2. **Fable MCP Server** — a new standalone Python package at `fable-mcp/` in the same repo root
|
|
||||||
|
|
||||||
A third sub-project (Forgejo MCP for CI/CD automation) is planned as a follow-on after the Fable MCP is working.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Sub-project 1: Fable API Key Feature
|
|
||||||
|
|
||||||
### Motivation
|
|
||||||
|
|
||||||
All existing Fable routes use session-based authentication (`session["user_id"]`). The MCP server runs as a local process and cannot maintain a browser session, so a stateless bearer token mechanism is required. API keys are user-scoped and carry a read/write permission level.
|
|
||||||
|
|
||||||
### Database
|
|
||||||
|
|
||||||
New table `api_keys` via migration `0027_add_api_keys.py`:
|
|
||||||
- `down_revision = "0026"` (the briefing tables migration)
|
|
||||||
- `revision = "0027"`
|
|
||||||
|
|
||||||
| Column | Type | Notes |
|
|
||||||
|--------|------|-------|
|
|
||||||
| `id` | Integer PK | |
|
|
||||||
| `user_id` | Integer FK → users | CASCADE delete |
|
|
||||||
| `name` | Text | Human-readable label |
|
|
||||||
| `key_hash` | Text | SHA-256 of the full key, used for lookup |
|
|
||||||
| `key_prefix` | Text | First 8 chars (e.g. `fmcp_ab12`), shown in UI |
|
|
||||||
| `scope` | Text | `"read"` or `"write"` |
|
|
||||||
| `last_used_at` | Timestamp | Nullable, updated on each authenticated request |
|
|
||||||
| `created_at` | Timestamp | |
|
|
||||||
| `revoked_at` | Timestamp | Nullable — soft delete |
|
|
||||||
|
|
||||||
Full keys are generated as `fmcp_<32 random url-safe chars>`, returned once at creation, never stored. Only the SHA-256 hash is persisted.
|
|
||||||
|
|
||||||
### Auth Middleware (`auth.py`)
|
|
||||||
|
|
||||||
`_check_auth` is updated to check the `Authorization: Bearer <token>` header before falling back to session auth:
|
|
||||||
|
|
||||||
1. If header present: hash the token, look up in `api_keys` where `revoked_at IS NULL`
|
|
||||||
2. If found: update `last_used_at`, set `g.user` and `g.api_key`
|
|
||||||
3. Scope enforcement: if `g.api_key.scope == "read"` and `request.method` is not `GET`, return 403
|
|
||||||
4. If header absent: existing session logic runs unchanged
|
|
||||||
|
|
||||||
Session auth is untouched — no regression risk for the web UI.
|
|
||||||
|
|
||||||
**Admin routes and API keys:** Routes decorated with `admin_required` check `user.role == "admin"` after auth resolves. API keys authenticate as the key owner — so a write-scoped key for a non-admin user will fail admin-protected routes with 403 (role check, not scope check). This is intentional: API keys cannot elevate privilege beyond the user's role.
|
|
||||||
|
|
||||||
### Routes (`/api/api-keys`)
|
|
||||||
|
|
||||||
New blueprint `api_keys_bp`, registered in `app.py`:
|
|
||||||
|
|
||||||
- `GET /api/api-keys` — list caller's keys (prefix, name, scope, last_used_at, created_at — never the hash or full key)
|
|
||||||
- `POST /api/api-keys` — create key; body: `{name, scope}`. Returns full key in response **once only**
|
|
||||||
- `DELETE /api/api-keys/:id` — revoke by setting `revoked_at = now()`
|
|
||||||
|
|
||||||
### Settings UI
|
|
||||||
|
|
||||||
New "API Keys" tab in `SettingsView.vue` (added to `VALID_TABS`):
|
|
||||||
|
|
||||||
- Create form: name input + read/write radio/toggle + "Generate Key" button
|
|
||||||
- After creation: one-time modal displaying the full key with a copy button and a warning that it will not be shown again
|
|
||||||
- Keys table: columns for name, scope badge, prefix, last used, revoke button
|
|
||||||
- Revoke shows inline confirmation before calling DELETE
|
|
||||||
|
|
||||||
### New Search Endpoint
|
|
||||||
|
|
||||||
`GET /api/search?q=<query>&content_type=note|task|all&limit=N`
|
|
||||||
|
|
||||||
Calls the existing `semantic_search_notes()` service (already in `services/embeddings.py`). The `content_type` parameter maps to the service's `is_task` argument:
|
|
||||||
- `content_type=note` → `is_task=False`
|
|
||||||
- `content_type=task` → `is_task=True`
|
|
||||||
- `content_type=all` (default) → `is_task=None`
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"results": [
|
|
||||||
{"id": 1, "title": "...", "body": "...", "is_task": false, "similarity": 0.87, "tags": [...]}
|
|
||||||
],
|
|
||||||
"total": 5
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
This endpoint is needed by the MCP's `search.py` tool module. It reuses existing infrastructure with no new ML work.
|
|
||||||
|
|
||||||
### Conversation Type: `"mcp"`
|
|
||||||
|
|
||||||
A new conversation type `"mcp"` is added alongside `"chat"` and `"briefing"`. This requires changes in two places in the main app:
|
|
||||||
|
|
||||||
**`services/chat.py`** — `create_conversation(user_id, title, model)` gains an optional `conversation_type: str = "chat"` parameter, passed through to the `Conversation` constructor.
|
|
||||||
|
|
||||||
**`routes/chat.py`** — `POST /api/chat/conversations` accepts an optional `conversation_type` body field (defaults to `"chat"`), passed to `create_conversation`. `GET /api/chat/conversations` continues to default-filter to `conversation_type="chat"` (excluding `"mcp"` and `"briefing"`); pass `?type=mcp` to retrieve MCP conversations.
|
|
||||||
|
|
||||||
**Retention:** `cleanup_old_conversations` in `routes/chat.py` currently deletes conversations regardless of type. It must be updated to exclude `conversation_type="mcp"` from the sweep, so MCP audit-trail conversations are not automatically pruned.
|
|
||||||
|
|
||||||
MCP conversations:
|
|
||||||
- Are created with a caller-supplied name (e.g., `"MCP Session 2026-03-23"`) or auto-named
|
|
||||||
- Are excluded from the default chat list
|
|
||||||
- Are accessible via `GET /api/chat/conversations?type=mcp` for inspection
|
|
||||||
- Appear in the Fable UI if explicitly navigated to, providing an audit trail of MCP-driven interactions
|
|
||||||
|
|
||||||
### Chat SSE Wire Format
|
|
||||||
|
|
||||||
The MCP's `chat.py` tool must consume the existing SSE stream. The relevant endpoints and event schema:
|
|
||||||
|
|
||||||
- **Create conversation:** `POST /api/chat/conversations` → `{id, title, ...}`
|
|
||||||
- **Post message + start generation:** `POST /api/chat/conversations/:id/messages` → `{message_id, ...}`; then connect to:
|
|
||||||
- **Stream:** `GET /api/chat/conversations/:id/stream` (SSE)
|
|
||||||
|
|
||||||
SSE event format (each line is `data: <json>`):
|
|
||||||
- `{"type": "token", "content": "..."}` — streaming token
|
|
||||||
- `{"type": "tool_call", "name": "...", "result": {...}}` — tool fired
|
|
||||||
- `{"type": "done", "content": "...", "tools_used": [...]}` — generation complete; this is the termination event
|
|
||||||
|
|
||||||
The MCP tool reads tokens until it receives `type: "done"`, then returns `response` (full content) and `tools_used` (list of tool names).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Sub-project 2: Fable MCP Server
|
|
||||||
|
|
||||||
### Location
|
|
||||||
|
|
||||||
`fable-mcp/` at the repository root, alongside `src/`, `frontend/`, `alembic/`. It is **not** part of the main Docker build and has no import relationship with `scribe`. It will be extracted to its own Forgejo repo once stable.
|
|
||||||
|
|
||||||
### Package Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
fable-mcp/
|
|
||||||
pyproject.toml # entry point: fable-mcp = "fable_mcp.server:main"
|
|
||||||
README.md
|
|
||||||
.env.example # FABLE_URL=http://localhost:8080, FABLE_API_KEY=fmcp_...
|
|
||||||
fable_mcp/
|
|
||||||
__init__.py
|
|
||||||
server.py # FastMCP instance, imports + registers all tool modules
|
|
||||||
client.py # FableClient: async httpx wrapper, env var config, FableAPIError
|
|
||||||
tools/
|
|
||||||
__init__.py
|
|
||||||
notes.py # list_notes, get_note, create_note, update_note, delete_note
|
|
||||||
tasks.py # list_tasks, get_task, create_task, update_task, delete_task,
|
|
||||||
# patch_task_status, add_task_log
|
|
||||||
projects.py # list_projects, get_project, create_project, update_project,
|
|
||||||
# delete_project, get_project_summary
|
|
||||||
milestones.py # list_milestones, get_milestone, create_milestone,
|
|
||||||
# update_milestone, delete_milestone
|
|
||||||
search.py # semantic_search — calls GET /api/search
|
|
||||||
chat.py # send_message — creates/continues conversation, returns response
|
|
||||||
```
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[project]
|
|
||||||
dependencies = [
|
|
||||||
"mcp[cli]>=1.0",
|
|
||||||
"httpx>=0.27",
|
|
||||||
"python-dotenv>=1.0",
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
### `client.py`
|
|
||||||
|
|
||||||
`FableClient` is an async context manager wrapping `httpx.AsyncClient`. It reads `FABLE_URL` and `FABLE_API_KEY` from environment variables (with `python-dotenv` fallback to `.env`). All methods are `async def` and return parsed dicts. A `FableAPIError(status_code, message)` exception is raised for any non-2xx response.
|
|
||||||
|
|
||||||
A module-level singleton `_client: FableClient` is initialized at server startup and shared across all tool modules.
|
|
||||||
|
|
||||||
### `server.py`
|
|
||||||
|
|
||||||
Uses `mcp[cli]`'s `FastMCP` class. Imports all tool modules, which register their tools via decorators against the shared `FastMCP` instance. Entry point `main()` calls `mcp.run()` (stdio transport).
|
|
||||||
|
|
||||||
### Tool Modules
|
|
||||||
|
|
||||||
Each module imports `_client` from `client.py` and defines tools as `async def` functions decorated with `@mcp.tool()`. Tool docstrings serve as the MCP tool descriptions visible to Claude.
|
|
||||||
|
|
||||||
**`notes.py`** tools:
|
|
||||||
- `list_notes(query, tags, project_id, limit, offset)`
|
|
||||||
- `get_note(note_id)`
|
|
||||||
- `create_note(title, body, tags, project_id)`
|
|
||||||
- `update_note(note_id, title, body, tags, project_id)`
|
|
||||||
- `delete_note(note_id)`
|
|
||||||
|
|
||||||
**`tasks.py`** tools:
|
|
||||||
- `list_tasks(query, project_id, milestone_id, status, priority, limit, offset)`
|
|
||||||
- `get_task(task_id)`
|
|
||||||
- `create_task(title, body, project_id, milestone_id, priority, status)`
|
|
||||||
- `update_task(task_id, title, body, project_id, milestone_id, priority, status)`
|
|
||||||
- `delete_task(task_id)`
|
|
||||||
- `patch_task_status(task_id, status)`
|
|
||||||
- `add_task_log(task_id, content)`
|
|
||||||
|
|
||||||
**`projects.py`** tools:
|
|
||||||
- `list_projects()`
|
|
||||||
- `get_project(project_id)` — includes milestone summary
|
|
||||||
- `create_project(title, description, goal, color)`
|
|
||||||
- `update_project(project_id, title, description, goal, status, color)`
|
|
||||||
- `delete_project(project_id)`
|
|
||||||
|
|
||||||
**`milestones.py`** tools:
|
|
||||||
- `list_milestones(project_id, status)`
|
|
||||||
- `get_milestone(project_id, milestone_id)`
|
|
||||||
- `create_milestone(project_id, title, description, order_index)`
|
|
||||||
- `update_milestone(project_id, milestone_id, title, description, status, order_index)`
|
|
||||||
- `delete_milestone(project_id, milestone_id)`
|
|
||||||
|
|
||||||
**`search.py`** tools:
|
|
||||||
- `semantic_search(query, content_type, limit)` — `content_type` is `"note"`, `"task"`, or `"all"` (avoids shadowing Python's `type` builtin). Calls `GET /api/search`, returns ranked results with similarity scores.
|
|
||||||
|
|
||||||
**`chat.py`** tools:
|
|
||||||
- `send_message(message, conversation_name)`:
|
|
||||||
- Looks up or creates a Fable conversation with the given name and `conversation_type="mcp"`
|
|
||||||
- Posts the message via `POST /api/chat/conversations/:id/messages`
|
|
||||||
- Consumes SSE stream from `GET /api/chat/conversations/:id/stream` until `type: "done"`
|
|
||||||
- Returns: `{response: str, tools_used: [str], conversation_id: int}`
|
|
||||||
- MCP conversations are excluded from the normal chat UI list
|
|
||||||
|
|
||||||
### Error Handling
|
|
||||||
|
|
||||||
`FableAPIError` is caught at each tool boundary and returned as a descriptive string rather than propagating as an exception. This ensures Claude receives a readable error message (e.g., `"Fable API error 403: Read-only key cannot perform write operations."`) rather than a stack trace. Network errors (`httpx.RequestError`) are similarly caught and surfaced as strings.
|
|
||||||
|
|
||||||
Scope enforcement lives entirely in Fable's auth middleware — the MCP does not duplicate it.
|
|
||||||
|
|
||||||
### Claude Code Registration
|
|
||||||
|
|
||||||
After `pip install -e fable-mcp/` (or `uv tool install ./fable-mcp`), add to `~/.claude/settings.json`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"mcpServers": {
|
|
||||||
"fable": {
|
|
||||||
"command": "fable-mcp",
|
|
||||||
"env": {
|
|
||||||
"FABLE_URL": "http://localhost:8080",
|
|
||||||
"FABLE_API_KEY": "fmcp_your_key_here"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Claude Code spawns the process over stdio automatically. No Docker, no daemon.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Build & Repo Plan
|
|
||||||
|
|
||||||
1. Implement and test within `scribe/fable-mcp/`
|
|
||||||
2. Once stable, extract to a new Forgejo repo (`bvandeusen/fable-mcp`)
|
|
||||||
3. Forgejo MCP (Gitea MCP) added as a second MCP server to automate build/push/config workflows — separate spec when ready
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Out of Scope (v1)
|
|
||||||
|
|
||||||
- MCP resources (browsable URI tree) — tools cover all use cases in Claude Code
|
|
||||||
- Forgejo MCP — follow-on spec
|
|
||||||
- Rate limiting on API key endpoints
|
|
||||||
- Key expiry / TTL
|
|
||||||
- Per-resource scope (e.g., key restricted to one project)
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,200 +0,0 @@
|
|||||||
# Speech-to-Speech (S2S) Design Spec
|
|
||||||
|
|
||||||
**Branch:** `feature/voice-s2s`
|
|
||||||
**Date:** 2026-03-29
|
|
||||||
**Status:** Approved for implementation
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Decisions
|
|
||||||
|
|
||||||
- **STT:** faster-whisper (in-process Python, model size configurable via `STT_MODEL` env var)
|
|
||||||
- **TTS:** Kokoro TTS (in-process Python, voice/speed/style configurable per-user)
|
|
||||||
- **Input mode:** Push-to-talk (Phase 1); VAD deferred
|
|
||||||
- **No browser STT/TTS fallbacks** — self-hosted only, data stays on-server
|
|
||||||
- **No Android app** — web only for this implementation
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## New Env Vars (`config.py`)
|
|
||||||
|
|
||||||
| Var | Default | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `VOICE_ENABLED` | `false` | Feature flag — opt-in |
|
|
||||||
| `STT_BACKEND` | `faster-whisper` | Only supported value currently |
|
|
||||||
| `STT_MODEL` | `base.en` | `tiny.en` / `base.en` / `small.en` / `medium.en` |
|
|
||||||
| `TTS_BACKEND` | `kokoro` | Only supported value currently |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Per-User Settings (stored in `settings` table)
|
|
||||||
|
|
||||||
| Key | Default | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `voice_tts_voice` | `af_heart` | Kokoro voice ID |
|
|
||||||
| `voice_tts_speed` | `1.0` | Speech rate, 0.7–1.3 |
|
|
||||||
| `voice_speech_style` | `conversational` | `conversational` / `concise` / `detailed` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## New Backend Files
|
|
||||||
|
|
||||||
### `src/scribe/services/stt.py`
|
|
||||||
Lazy singleton `WhisperModel` loader. Public API:
|
|
||||||
- `load_stt_model()` — called at startup via `asyncio.create_task`
|
|
||||||
- `transcribe(audio_bytes, mime_type) -> str` — runs in `run_in_executor`; writes bytes to `NamedTemporaryFile`, returns concatenated segment text
|
|
||||||
- `stt_available() -> bool`
|
|
||||||
|
|
||||||
### `src/scribe/services/tts.py`
|
|
||||||
Lazy singleton `KPipeline` loader. Public API:
|
|
||||||
- `load_tts_model()` — called at startup
|
|
||||||
- `synthesise(text, voice, speed) -> bytes` — runs in `run_in_executor`; returns WAV bytes (24kHz, 16-bit mono)
|
|
||||||
- `list_voices() -> list[dict]` — returns static list of known Kokoro voice IDs + labels
|
|
||||||
- `tts_available() -> bool`
|
|
||||||
|
|
||||||
### `src/scribe/routes/voice.py`
|
|
||||||
Blueprint at `/api/voice`, all routes `@login_required`.
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `/api/voice/status` | GET | STT/TTS availability; `enabled` false if `VOICE_ENABLED=false` |
|
|
||||||
| `/api/voice/voices` | GET | List available Kokoro voices |
|
|
||||||
| `/api/voice/transcribe` | POST | multipart `audio` field → `{"transcript": "...", "duration_ms": 123}` |
|
|
||||||
| `/api/voice/synthesise` | POST | `{"text", "voice", "speed"}` → WAV bytes |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Modified Backend Files
|
|
||||||
|
|
||||||
### `src/scribe/app.py`
|
|
||||||
- Register `voice_bp` blueprint
|
|
||||||
- In `startup()`: `asyncio.create_task(load_stt_model())` + `asyncio.create_task(load_tts_model())` when `VOICE_ENABLED`
|
|
||||||
|
|
||||||
### `src/scribe/config.py`
|
|
||||||
- Add 4 new env var attributes
|
|
||||||
- Add validation in `validate()`
|
|
||||||
|
|
||||||
### `src/scribe/services/llm.py`
|
|
||||||
- Add `voice_mode: bool = False` and `voice_speech_style: str = "conversational"` to `build_context()`
|
|
||||||
- When `voice_mode=True`, prepend: *"Respond naturally as if speaking aloud. No markdown, bullet points, headers, or code blocks. Complete sentences only."*
|
|
||||||
- Append style modifier based on `voice_speech_style`
|
|
||||||
|
|
||||||
### `src/scribe/services/generation_task.py`
|
|
||||||
- Add `voice_mode: bool = False` to `run_generation()`
|
|
||||||
- Read `voice_speech_style` from settings when voice_mode; pass both to `build_context()`
|
|
||||||
|
|
||||||
### `src/scribe/routes/chat.py`
|
|
||||||
- Allow `"voice"` in `conversation_type` whitelist
|
|
||||||
|
|
||||||
### `src/scribe/services/chat.py`
|
|
||||||
- Exclude `conversation_type == "voice"` from auto-cleanup retention
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## New Frontend Files
|
|
||||||
|
|
||||||
### `frontend/src/composables/useVoiceRecorder.ts`
|
|
||||||
Wraps `MediaRecorder`. Exports: `recording`, `error`, `isSupported`, `startRecording()`, `stopRecording() -> Promise<Blob>`.
|
|
||||||
|
|
||||||
### `frontend/src/composables/useVoiceAudio.ts`
|
|
||||||
Wraps `AudioContext`. Exports: `playing`, `isSupported`, `play(blob)`, `stop()`.
|
|
||||||
|
|
||||||
### `frontend/src/components/VoiceOverlay.vue`
|
|
||||||
Floating PTT button (fixed bottom-right). Creates/reuses a `"voice"` conversation. Full flow: record → transcribe → send → stream → synthesise → play. Space bar hotkey (from `App.vue`). Mounted globally in `App.vue`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Modified Frontend Files
|
|
||||||
|
|
||||||
### `frontend/src/api/client.ts`
|
|
||||||
Add: `transcribeAudio(blob)`, `synthesiseSpeech(text, voice?, speed?)`, `getVoiceStatus()`, `getVoiceList()`
|
|
||||||
|
|
||||||
### `frontend/src/views/BriefingView.vue`
|
|
||||||
- "Listen" button: reads latest assistant message aloud via TTS
|
|
||||||
- Mic button in input bar: PTT → transcribe → auto-fill input → send
|
|
||||||
- Auto-TTS on assistant response when in listen mode
|
|
||||||
|
|
||||||
### `frontend/src/views/SettingsView.vue`
|
|
||||||
- New "Voice" tab: voice dropdown, speed slider, speech style radio
|
|
||||||
- Loads from `/api/settings`, saves via `PUT /api/settings`
|
|
||||||
|
|
||||||
### `frontend/src/App.vue`
|
|
||||||
- Mount `<VoiceOverlay />`
|
|
||||||
- Space bar → `"voice:ptt-toggle"` custom event
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Audio Format
|
|
||||||
|
|
||||||
| Direction | Format | Rationale |
|
|
||||||
|---|---|---|
|
|
||||||
| Browser → Server | WebM/Opus | Native `MediaRecorder` output; no re-encoding |
|
|
||||||
| Server → Browser | WAV (24kHz, 16-bit mono) | Kokoro native; no re-encoding; `decodeAudioData` compatible |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Dependencies to Add (`pyproject.toml`)
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[project.optional-dependencies]
|
|
||||||
voice = [
|
|
||||||
"faster-whisper>=1.0",
|
|
||||||
"kokoro>=0.9",
|
|
||||||
"soundfile>=0.12",
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
Install unconditionally in Docker (activated by `VOICE_ENABLED` at runtime):
|
|
||||||
```dockerfile
|
|
||||||
RUN pip install faster-whisper kokoro soundfile
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Database Migration
|
|
||||||
|
|
||||||
No schema changes required. `conversation_type` is unconstrained TEXT. Voice settings use existing key-value `settings` table. Optional no-op migration `0034_voice_conversation_type.py` for audit trail.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation Phases
|
|
||||||
|
|
||||||
### Phase 1 — Backend services + routes
|
|
||||||
1. Add env vars to `config.py`
|
|
||||||
2. Create `services/stt.py` (faster-whisper)
|
|
||||||
3. Create `services/tts.py` (Kokoro)
|
|
||||||
4. Create `routes/voice.py` (4 endpoints)
|
|
||||||
5. Wire model loading into `app.py` startup
|
|
||||||
6. Add `voice_mode` to `build_context()` + `run_generation()`
|
|
||||||
7. Allow `"voice"` conversation type in chat route + cleanup exclusion
|
|
||||||
|
|
||||||
### Phase 2 — BriefingView listen + voice follow-up
|
|
||||||
1. Create `useVoiceRecorder.ts`
|
|
||||||
2. Create `useVoiceAudio.ts`
|
|
||||||
3. Add voice API functions to `client.ts`
|
|
||||||
4. Add "Listen" button + mic button to `BriefingView.vue`
|
|
||||||
|
|
||||||
### Phase 3 — VoiceOverlay for general voice chat
|
|
||||||
1. Create `VoiceOverlay.vue`
|
|
||||||
2. Mount in `App.vue` + Space bar hotkey
|
|
||||||
|
|
||||||
### Phase 4 — Settings UI
|
|
||||||
1. Add "Voice" tab to `SettingsView.vue`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Kokoro Voice Reference
|
|
||||||
|
|
||||||
| ID | Character |
|
|
||||||
|---|---|
|
|
||||||
| `af_heart` | American female, warm (recommended default) |
|
|
||||||
| `af_bella` | American female, expressive |
|
|
||||||
| `af_nicole` | American female, breathy/intimate |
|
|
||||||
| `af_sarah` | American female, clear |
|
|
||||||
| `af_sky` | American female, bright |
|
|
||||||
| `am_adam` | American male, neutral |
|
|
||||||
| `am_michael` | American male, deeper |
|
|
||||||
| `bf_emma` | British female |
|
|
||||||
| `bf_isabella` | British female, formal |
|
|
||||||
| `bm_george` | British male |
|
|
||||||
| `bm_lewis` | British male, casual |
|
|
||||||
@@ -1,216 +0,0 @@
|
|||||||
# Agentic Briefing — Design Spec
|
|
||||||
|
|
||||||
**Date:** 2026-04-10
|
|
||||||
**Status:** Proposed
|
|
||||||
**Author:** bvandeusen + Claude
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
The current briefing pipeline hallucinates calendar events, tasks, and news items that do not exist in the user's actual data. Observed examples from production:
|
|
||||||
|
|
||||||
- A morning briefing asserting "your dentist appointment is still in progress" when no such event existed
|
|
||||||
- An 8am check-in mentioning "a quick meeting at 10:30 AM" with no backing calendar entry
|
|
||||||
- A midday check-in inventing "a team huddle at 2:30 PM" and "the Q2 budget draft due by Friday"
|
|
||||||
- The model calling `search_images` in response to a clarifying question about a fabricated meeting
|
|
||||||
|
|
||||||
These are not bugs in data retrieval — the data gathering code (`_gather_internal`, `_gather_external`) returns correct values. They are a structural consequence of how the briefing context is assembled.
|
|
||||||
|
|
||||||
## Root cause — the "no receipt" problem
|
|
||||||
|
|
||||||
The current `run_compilation` pipeline does this:
|
|
||||||
|
|
||||||
1. Python code gathers data (tasks, events, weather, news) from the database
|
|
||||||
2. Python formats the data into a structured text blob as the user-role message: `TODAY'S EVENTS: ...`, `DUE TODAY: ...`, etc.
|
|
||||||
3. The LLM is called **once** with `[system_prompt, user_prompt]` and produces prose
|
|
||||||
4. Only the prose reply is written to the conversation. **The underlying data is never persisted in the conversation history.**
|
|
||||||
|
|
||||||
When the user later chats in the briefing conversation, the chat endpoint loads the full conversation history — which contains the model's *prose* from earlier but not the data that prose was derived from. The model's own prior output becomes the only source of "truth" available for follow-up questions. If that prose asserted a fact (real or hallucinated), the model has no way to distinguish it from ground truth when generating the next reply, and it will double down.
|
|
||||||
|
|
||||||
Compounding factors:
|
|
||||||
|
|
||||||
- **Empty sections are silently omitted.** If `calendar_events` is empty, the user prompt contains no `TODAY'S EVENTS:` line at all. The model has no explicit "zero events" signal — combined with an imperative system prompt ("note calendar events and tasks"), it interprets the silence as "I should mention some" and fabricates.
|
|
||||||
- **Scheduled slot injections append synthetic turns.** `run_slot_injection` writes a fake `[Midday briefing update]` user message and the assistant reply into the persistent conversation. By evening, the chat history contains three separate briefings, each potentially with errors, all treated as equal-weight context on follow-up.
|
|
||||||
- **The `search_images` tool is available during briefing chat**, with a negative-instruction description ("Not for factual questions"). Small and mid-sized models frequently ignore negative guidance in tool descriptions and call the tool anyway.
|
|
||||||
|
|
||||||
## Solution — agentic briefing (the receipt model)
|
|
||||||
|
|
||||||
Replace the one-shot synthesis with a tool-call loop. The briefing is no longer "a text blob synthesized from pre-gathered data." It is **a scheduled agent run**: the LLM is given a system prompt, a curated set of read-only data tools, and a trigger ("generate the morning briefing"). The model must call tools to see what exists. Every tool call and tool result becomes part of the conversation history, where it lives as a permanent, structured receipt.
|
|
||||||
|
|
||||||
### Why this fixes the hallucination
|
|
||||||
|
|
||||||
When the model calls `list_events(today) → []`, that empty array is now a persistent message in the conversation. On a follow-up question like "what meeting?", the chat endpoint loads the conversation and the model sees its own tool result from the morning showing no events. Answering "you had a meeting at 10:30" would require the model to directly contradict a tool result sitting two messages back — something LLMs are much more reliable at avoiding than contradicting their own prior prose.
|
|
||||||
|
|
||||||
In short: **the model cannot fabricate what has a visible receipt proving it does not exist.**
|
|
||||||
|
|
||||||
### What the model sees — before vs after
|
|
||||||
|
|
||||||
**Before** — the model sees one structured blob and produces prose. The blob is discarded.
|
|
||||||
|
|
||||||
```
|
|
||||||
[system] You are a personal assistant... weave together what matters...
|
|
||||||
[user] Date: 2026-04-10
|
|
||||||
WEATHER: Home — partly cloudy, 8–16°C
|
|
||||||
DUE TODAY: Fix briefing hallucinations
|
|
||||||
OVERDUE (2 tasks): ...
|
|
||||||
[assistant] Morning! Looks like a quiet day with two overdue items...
|
|
||||||
```
|
|
||||||
|
|
||||||
**After** — the model is given tools and iterates to ground truth.
|
|
||||||
|
|
||||||
```
|
|
||||||
[system] You are the user's personal assistant giving their morning briefing.
|
|
||||||
Use tools to see what's relevant. Only mention things you learned from
|
|
||||||
tool calls. If a tool returns nothing, say so honestly.
|
|
||||||
[user] Generate the morning briefing.
|
|
||||||
[assistant] tool_call: list_tasks(filter="due_today")
|
|
||||||
[tool] [{"id": 81, "title": "Fix briefing hallucinations", ...}]
|
|
||||||
[assistant] tool_call: list_tasks(filter="overdue")
|
|
||||||
[tool] [{"id": 42, ...}, {"id": 43, ...}]
|
|
||||||
[assistant] tool_call: list_events(today)
|
|
||||||
[tool] []
|
|
||||||
[assistant] tool_call: get_weather(location="home")
|
|
||||||
[tool] {"description": "partly cloudy", "temp_min": 8, "temp_max": 16}
|
|
||||||
[assistant] tool_call: get_rss_items(max=8)
|
|
||||||
[tool] [...]
|
|
||||||
[assistant] Morning! Partly cloudy today, 8 to 16 — nothing on the calendar
|
|
||||||
so it's a clean run at the desk. Two things to keep in mind...
|
|
||||||
```
|
|
||||||
|
|
||||||
The conversation now contains verifiable receipts: `list_events(today)` returned `[]`, and that result sits in context forever (until pruned). Follow-up questions operate against those receipts.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Existing infrastructure (reused, not rebuilt)
|
|
||||||
|
|
||||||
The codebase already has the agentic primitives — they're used for regular chat:
|
|
||||||
|
|
||||||
- **`llm.py::stream_chat_with_tools`** — the streaming tool-use loop that talks to Ollama with a `tools` parameter and yields tool-call chunks
|
|
||||||
- **`generation_task.py::_stream_with_retry`** — wraps `stream_chat_with_tools` with retry-on-500 behavior for cold-model races
|
|
||||||
- **`tools.py`** — defines 40+ tool schemas and an execution dispatcher
|
|
||||||
|
|
||||||
The briefing bypasses all of this and calls a one-shot `_llm_synthesise` helper. The refactor is mainly "route briefings through the same pipeline regular chat already uses."
|
|
||||||
|
|
||||||
### New modules
|
|
||||||
|
|
||||||
**`briefing_tools.py`** — a small wrapper exposing a curated read-only subset of `tools.py` for briefing runs. This is an **explicit allowlist**, not a blocklist, so newly-added tools must be opted in:
|
|
||||||
|
|
||||||
| Tool | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `list_tasks` (with filter args) | See what's actionable today, overdue, high priority |
|
|
||||||
| `list_events` (today / upcoming) | Know what's on the calendar |
|
|
||||||
| `get_weather` | Current/forecast weather |
|
|
||||||
| `get_rss_items` | Pull news themes filtered by user preferences |
|
|
||||||
| `list_projects` | Understand project context |
|
|
||||||
| `search_projects` | Surface active project summaries |
|
|
||||||
| `list_notes` (recent) | Capture follow-ups from yesterday |
|
|
||||||
|
|
||||||
**Explicitly omitted** from the briefing tool set:
|
|
||||||
|
|
||||||
- `search_images`, `search_web`, `research_topic`, `read_article` — external search is not a briefing concern, and `search_images` is the source of the "Peter Kyle Science Secretary" image-search bug
|
|
||||||
- All `create_*`, `update_*`, `delete_*` — briefings are read-only; a scheduled background job must not decide to mutate the user's data on its own
|
|
||||||
- `set_rag_scope`, `calculate` — not relevant to briefing content
|
|
||||||
|
|
||||||
### New briefing function
|
|
||||||
|
|
||||||
**`briefing_pipeline.py::run_agentic_briefing(user_id, slot, model, conversation_id)`** replaces `run_compilation`'s body (and eventually `run_slot_injection`). Internally:
|
|
||||||
|
|
||||||
1. Build a slot-specific system prompt (see below)
|
|
||||||
2. Load the curated briefing tools from `briefing_tools.py`
|
|
||||||
3. Seed messages with `[system, user]` where the user message is a simple trigger like `"Generate the morning briefing."`
|
|
||||||
4. Enter a tool-call loop (max 8 iterations):
|
|
||||||
- Call `stream_chat_with_tools`
|
|
||||||
- If the model returns `tool_calls`, execute them via the existing dispatcher, append tool results, continue
|
|
||||||
- If the model returns a final assistant message with no pending tool calls, break
|
|
||||||
5. Return `(final_prose, full_message_list, metadata)`
|
|
||||||
|
|
||||||
The full message list is important: it's written to the conversation along with the final prose, so the tool-call receipts become part of the persistent record.
|
|
||||||
|
|
||||||
### Slot-specific system prompts
|
|
||||||
|
|
||||||
Compilation (full morning briefing):
|
|
||||||
|
|
||||||
```
|
|
||||||
You are the user's personal assistant giving their full morning briefing.
|
|
||||||
Use the tools available to see what's actually relevant today — tasks due,
|
|
||||||
overdue items, events on the calendar, weather, news themes, project state
|
|
||||||
— and weave it into a warm, natural-sounding summary.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
- Call tools to see the data. Never assert facts you didn't learn from a tool.
|
|
||||||
- If a tool returns nothing (no events today, no overdue tasks), say so
|
|
||||||
honestly. Don't fabricate items to fill space.
|
|
||||||
- Write flowing prose. No markdown, no headers, no bullet points.
|
|
||||||
- Aim for 6–10 sentences. Skip topics that have nothing interesting.
|
|
||||||
- Close on one or two concrete, actionable suggestions.
|
|
||||||
|
|
||||||
User profile (for tone and preferences):
|
|
||||||
{profile_body}
|
|
||||||
```
|
|
||||||
|
|
||||||
Check-ins (midday, afternoon):
|
|
||||||
|
|
||||||
```
|
|
||||||
You are the user's personal assistant giving a brief {slot} check-in.
|
|
||||||
Use tools to see what's changed since this morning. Focus on progress
|
|
||||||
and what's still unaddressed.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
- Call tools to see current state. Never assert facts without tool results.
|
|
||||||
- If nothing meaningful has changed, say so briefly — don't invent progress.
|
|
||||||
- 3–5 sentences, natural prose, no markdown.
|
|
||||||
```
|
|
||||||
|
|
||||||
### Conversation hygiene — removing the fake user messages
|
|
||||||
|
|
||||||
The current scheduler appends two fake messages on every slot injection:
|
|
||||||
|
|
||||||
```python
|
|
||||||
await post_message(conv.id, "user", f"[{slot.title()} briefing update]")
|
|
||||||
await post_message(conv.id, "assistant", text)
|
|
||||||
```
|
|
||||||
|
|
||||||
Under the agentic model, we drop the fake `"user"` message entirely. Slot updates become plain assistant messages tagged with `metadata.briefing_slot = "midday"`. The chat endpoint's message loader is updated to filter these when building the LLM context on follow-ups, so a user chatting in the briefing conversation doesn't see three earlier briefings smashed into their history. The UI continues to show them as visible timeline entries.
|
|
||||||
|
|
||||||
(This is the Option A filter from the earlier discussion — a small, surgical change compared to migrating briefings to a separate sidecar table. Sidecar storage remains a possible future step.)
|
|
||||||
|
|
||||||
### Ollama setup compatibility
|
|
||||||
|
|
||||||
The existing Ollama deployment uses non-parallel mode across two GPUs, with a concern about context duplication between cards. This is the correct setup for agentic briefings:
|
|
||||||
|
|
||||||
- Each tool-call iteration shares the same KV cache on the same GPU, so appending tool results is cheap
|
|
||||||
- There is no race where a second iteration could land on a different card with a different cache state
|
|
||||||
- The trade-off is serialization: a briefing in progress will block a concurrent user chat request until it finishes, but scheduled briefings are rare enough (4× per day) that this is acceptable
|
|
||||||
|
|
||||||
If GPU contention becomes a problem later, the right lever is pinning specific models to specific GPUs (e.g., background tasks on GPU 1, interactive models on GPU 0) — not enabling parallelism.
|
|
||||||
|
|
||||||
## Cost & trade-offs
|
|
||||||
|
|
||||||
- **Latency:** a briefing now makes 5–7 inference calls (one per tool-call decision plus the final prose) instead of 1. On a local Ollama with a 7B model, expect 15–40s per briefing vs the current 5–10s. Acceptable for a background job; if it becomes painful at the user-facing check-in slots, investigate letting the model batch independent tool calls in a single turn (Ollama supports this).
|
|
||||||
- **Model requirements:** the default model must be reliable at tool calling. `qwen2.5:7b`, `llama3.1:8b`, `mistral-small-3:24b`, and similar handle it well. Models in the ≤3B class typically fail — they either emit no tool calls and return empty prose, or hallucinate invalid tool arguments. If a user's `default_model` is too small, agentic mode should fall back to legacy mode with a log warning.
|
|
||||||
- **Context growth:** tool results bloat the conversation. At 8 tool calls per compilation × 4 slots per day, a daily briefing conversation can reach 20-30 KB of JSON-heavy history. Fine for a day; aged briefings should be archived/rolled up after 7 days.
|
|
||||||
- **Tool subset drift:** someone adds a new mutating tool to `tools.py` and forgets to update the briefing allowlist. Mitigated by the allowlist model — the default for new tools is "not exposed to briefings."
|
|
||||||
- **Infinite loop safety:** a buggy model could tool-call forever. Hard cap at 8 iterations, log a warning if hit, return whatever prose was last produced (or a fallback message).
|
|
||||||
|
|
||||||
## Migration path
|
|
||||||
|
|
||||||
Ship incrementally, each step independently reversible:
|
|
||||||
|
|
||||||
**PR 1 — Agentic compilation behind a feature flag.** Add `briefing_tools.py`, add `run_agentic_briefing`, add a per-user setting `briefing_mode: "legacy" | "agentic"` (default legacy). Route only the 4am compilation through the new path when the flag is set. Keep slot-injection on the legacy path. Enable the flag on the author's account first, validate output quality over several days, then flip the default for all users. No DB migration required — the setting lives in the existing `settings` table.
|
|
||||||
|
|
||||||
**PR 2 — Agentic slot injections + conversation hygiene.** Migrate midday/afternoon check-ins to the same pipeline. Remove the fake `[Midday briefing update]` user-role message; slot updates become plain assistant messages tagged with `metadata.briefing_slot`. Add a chat-message-loader filter that excludes slot-tagged messages from the LLM context on follow-ups (they remain visible in the UI).
|
|
||||||
|
|
||||||
**PR 3 — UI polish.** Collapsed tool-call status row in the briefing card ("✓ checked calendar · ✓ looked at tasks · ✓ pulled weather"), expanding to show tool results on click. Tool-result cards (weather, news, task list) rendered inline where useful.
|
|
||||||
|
|
||||||
**PR 4 (optional) — Sidecar storage for briefing snapshots.** If the chat-filter approach in PR 2 feels too hacky, migrate briefings out of the conversations table and into a dedicated `briefing_snapshots` table. Frontend renders them as pinned timeline cards separate from chat. Larger refactor; defer until PR 1–3 prove the approach works.
|
|
||||||
|
|
||||||
## Secondary win — tightening the main chat system prompt
|
|
||||||
|
|
||||||
The regular chat is already agentic (it uses `stream_chat_with_tools`), but its system prompt does not explicitly require the model to ground factual claims in tool results. The prompt discipline introduced for briefings — *"Never assert facts you didn't learn from a tool. If a tool returns nothing, say so honestly."* — is worth applying to the main chat system prompt in a follow-up PR. The mechanism already works; only the framing needs tightening.
|
|
||||||
|
|
||||||
## Out of scope
|
|
||||||
|
|
||||||
- The Android Flutter client's failure to render `search_images` tool-result cards. This is a separate rendering gap in the mobile app and does not affect the server-side fix. Tracked separately.
|
|
||||||
- Re-evaluating the Ollama parallelism setting. The current non-parallel config is correct for this work.
|
|
||||||
- Replacing the background model for title/summary/observation extraction. That model's role is unrelated to briefings.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# act_runner configuration.
|
|
||||||
# Place this file at: /nfs/data/fabledsword/act_runner/config.yml
|
|
||||||
# It is bind-mounted read-only into the runner container.
|
|
||||||
|
|
||||||
log:
|
|
||||||
level: info
|
|
||||||
|
|
||||||
runner:
|
|
||||||
capacity: 2 # max concurrent jobs
|
|
||||||
timeout: 30m
|
|
||||||
# Maps runs-on labels to Docker images for job containers.
|
|
||||||
# Change this (and delete /data/.runner) to switch the base image.
|
|
||||||
labels:
|
|
||||||
- "py3.12-node22:docker://git.fabledsword.com/bvandeusen/runner-base:py3.12-node22"
|
|
||||||
|
|
||||||
cache:
|
|
||||||
enabled: true
|
|
||||||
dir: /data/.cache
|
|
||||||
|
|
||||||
container:
|
|
||||||
# Job containers use bridge networking — they get outbound internet access
|
|
||||||
# (needed to pull npm/pip packages and push to the registry) but are isolated
|
|
||||||
# from each other and the host.
|
|
||||||
network: bridge
|
|
||||||
|
|
||||||
# Paths the runner is allowed to bind-mount into job containers.
|
|
||||||
# The Docker socket is required for the build job (docker buildx).
|
|
||||||
valid_volumes:
|
|
||||||
- /var/run/docker.sock
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Forgejo Actions runner — deploy as a standalone Portainer stack.
|
|
||||||
# The runner connects outbound to Forgejo to receive jobs; no inbound ports needed.
|
|
||||||
# /var/run/docker.sock is mounted so the build job can run Docker Buildx.
|
|
||||||
#
|
|
||||||
# Setup: see the walkthrough at infra/RUNNER_SETUP.md
|
|
||||||
|
|
||||||
services:
|
|
||||||
act_runner:
|
|
||||||
image: gitea/act_runner:latest
|
|
||||||
restart: unless-stopped
|
|
||||||
environment:
|
|
||||||
- GITEA_INSTANCE_URL=https://git.fabledsword.com
|
|
||||||
- GITEA_RUNNER_REGISTRATION_TOKEN=${RUNNER_TOKEN}
|
|
||||||
- GITEA_RUNNER_NAME=swarm-runner
|
|
||||||
# Maps the runs-on: ubuntu-latest label used in workflow files to a Docker image.
|
|
||||||
- CONFIG_FILE=/config/config.yml
|
|
||||||
volumes:
|
|
||||||
- /var/run/docker.sock:/var/run/docker.sock
|
|
||||||
- act_runner_data:/data
|
|
||||||
- /nfs/data/fabledsword/act_runner/config.yml:/config/config.yml:ro
|
|
||||||
# No network needed — runner only makes outbound HTTPS connections to Forgejo.
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
act_runner_data:
|
|
||||||
Reference in New Issue
Block a user