Compare commits

...

8 Commits

Author SHA1 Message Date
bvandeusen f146485df3 feat: hot-reload voice models without server restart
Voice enabled/STT model are now DB-backed (admin settings), not env
vars. Added reload_stt_model()/reload_tts_model() that clear singletons
under lock and re-trigger loading. POST /api/admin/voice/reload triggers
both in background tasks. Settings UI polls /api/voice/status every 2.5s
until models are ready, with spinner feedback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:52:07 -04:00
bvandeusen eaf70500b8 feat: move voice enable/model config to admin UI
Replace VOICE_ENABLED env var gate with DB-backed admin setting.

- services/voice_config.py: reads voice_enabled + voice_stt_model from
  admin user's settings row (falls back to env var defaults)
- routes/admin.py: GET/PUT /api/admin/voice for admin configuration
- routes/voice.py, services/stt.py, services/tts.py: read enabled/model
  from DB via voice_config instead of Config directly
- app.py: always schedule model loaders at startup; they self-gate on
  the DB setting so no conditional needed at the call site
- SettingsView.vue: Voice section in Admin → Config tab (enable toggle +
  STT model dropdown); user Voice tab now points to admin panel when disabled

No env var required to test — enable via Settings → Admin → Config → Voice.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:22:18 -04:00
bvandeusen 6f84d90dff feat: voice S2S — faster-whisper STT, Kokoro TTS, PTT overlay
Implements full speech-to-speech pipeline (all 4 phases):

Backend (Phase 1):
- services/stt.py: lazy WhisperModel singleton, run_in_executor transcription
- services/tts.py: lazy KPipeline singleton, WAV synthesis at 24kHz/16-bit
- routes/voice.py: /api/voice/status, /voices, /transcribe, /synthesise
- config.py: VOICE_ENABLED, STT_BACKEND, STT_MODEL, TTS_BACKEND env vars
- app.py: load STT/TTS models at startup when VOICE_ENABLED=true
- llm.py: voice_mode + voice_speech_style params inject speak-naturally prefix
- generation_task.py: voice_mode passed through from chat route
- chat.py: "voice" conversation type allowed + excluded from retention cleanup
- pyproject.toml + Dockerfile: faster-whisper, kokoro, soundfile dependencies

Frontend (Phases 2–4):
- composables/useVoiceRecorder.ts: MediaRecorder PTT wrapper
- composables/useVoiceAudio.ts: AudioContext WAV playback wrapper
- BriefingView.vue: Listen button (TTS read-aloud), auto-TTS mode, mic PTT
- VoiceOverlay.vue: global floating PTT button; creates/reuses voice conv;
  full record→transcribe→stream→TTS flow; Space bar hold-to-talk via App.vue
- SettingsView.vue: Voice tab (status badge, speech style, voice/speed)
- App.vue: mounts VoiceOverlay; Space keydown/keyup fires voice:ptt-toggle
- api/client.ts: getVoiceStatus, getVoiceList, transcribeAudio, synthesiseSpeech

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:03:38 -04:00
bvandeusen 3581cc1582 docs: add voice S2S design spec
Covers STT (faster-whisper), TTS (Kokoro), per-user settings,
all new/modified files, audio format decisions, and 4-phase
implementation plan.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 18:17:58 -04:00
bvandeusen 3dd879640a chore(fable-mcp): bump version to 0.2.0
Reflects all changes since initial 0.1.0 release: RSS tools, task log
content/body fix, project and milestone status on create, and various
other fixes. Auto-bump hook will handle patch increments from here.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 16:01:12 -04:00
bvandeusen 0cbeb6b7ac fix: honour status param on project and milestone creation
create_project and create_milestone hardcoded status="active" and
ignored any value passed by the MCP or API callers. Route, service,
and model construction now all thread the status field through.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:19:50 -04:00
bvandeusen 218f946e48 fix: fable_add_task_log sends content not body to match API
The MCP tool was sending {"body": ...} but the task logs API route
expects {"content": ...}, causing 400 errors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:16:13 -04:00
bvandeusen 00643c778e security: fix 10 vulnerabilities from security audit
- SSRF: block private/internal URLs in image cache fetch
- SSRF: block private/internal URLs in RSS feed fetch (scheme guard)
- SSRF: block private/internal URLs in CalDAV URL setting
- Auth: require login for GET /api/images/<id> (was unauthenticated)
- Auth: restrict Ollama model pull/delete to admin users only
- Info disclosure: remove email from /api/users/search response
- OAuth: skip email-based account linking when email_verified is false
- Config: raise hard error on default SECRET_KEY when SECURE_COOKIES=true
- Rate limit: document proxy header requirement; add startup warning
- XSS: remove src/alt from global DOMPurify ADD_ATTR allowlist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 00:37:13 -04:00
40 changed files with 2051 additions and 33 deletions
+2
View File
@@ -13,6 +13,8 @@ WORKDIR /app
COPY pyproject.toml .
COPY src/ src/
RUN pip install --no-cache-dir .
# Voice dependencies (faster-whisper, Kokoro TTS, soundfile) — activated at runtime via VOICE_ENABLED
RUN pip install --no-cache-dir faster-whisper kokoro soundfile
# Build the fable-mcp wheel so it can be served for download
COPY fable-mcp/ fable-mcp/
+1 -1
View File
@@ -15,7 +15,7 @@ services:
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-fabled}:${POSTGRES_PASSWORD:-fabled}@db:5432/${POSTGRES_DB:-fabledassistant}"
OLLAMA_URL: "http://ollama:11434"
OLLAMA_MODEL: "${OLLAMA_MODEL:-llama3.1}"
OLLAMA_MODEL: "${OLLAMA_MODEL:-qwen3:8B}"
SECRET_KEY: "${SECRET_KEY:-dev-secret-change-me}"
# Uncomment and set to enable web research and image search via SearXNG:
# SEARXNG_URL: "http://searxng:8080"
+200
View File
@@ -0,0 +1,200 @@
# Speech-to-Speech (S2S) Design Spec
**Branch:** `feature/voice-s2s`
**Date:** 2026-03-29
**Status:** Approved for implementation
---
## Decisions
- **STT:** faster-whisper (in-process Python, model size configurable via `STT_MODEL` env var)
- **TTS:** Kokoro TTS (in-process Python, voice/speed/style configurable per-user)
- **Input mode:** Push-to-talk (Phase 1); VAD deferred
- **No browser STT/TTS fallbacks** — self-hosted only, data stays on-server
- **No Android app** — web only for this implementation
---
## New Env Vars (`config.py`)
| Var | Default | Description |
|---|---|---|
| `VOICE_ENABLED` | `false` | Feature flag — opt-in |
| `STT_BACKEND` | `faster-whisper` | Only supported value currently |
| `STT_MODEL` | `base.en` | `tiny.en` / `base.en` / `small.en` / `medium.en` |
| `TTS_BACKEND` | `kokoro` | Only supported value currently |
---
## Per-User Settings (stored in `settings` table)
| Key | Default | Description |
|---|---|---|
| `voice_tts_voice` | `af_heart` | Kokoro voice ID |
| `voice_tts_speed` | `1.0` | Speech rate, 0.71.3 |
| `voice_speech_style` | `conversational` | `conversational` / `concise` / `detailed` |
---
## New Backend Files
### `src/fabledassistant/services/stt.py`
Lazy singleton `WhisperModel` loader. Public API:
- `load_stt_model()` — called at startup via `asyncio.create_task`
- `transcribe(audio_bytes, mime_type) -> str` — runs in `run_in_executor`; writes bytes to `NamedTemporaryFile`, returns concatenated segment text
- `stt_available() -> bool`
### `src/fabledassistant/services/tts.py`
Lazy singleton `KPipeline` loader. Public API:
- `load_tts_model()` — called at startup
- `synthesise(text, voice, speed) -> bytes` — runs in `run_in_executor`; returns WAV bytes (24kHz, 16-bit mono)
- `list_voices() -> list[dict]` — returns static list of known Kokoro voice IDs + labels
- `tts_available() -> bool`
### `src/fabledassistant/routes/voice.py`
Blueprint at `/api/voice`, all routes `@login_required`.
| Endpoint | Method | Description |
|---|---|---|
| `/api/voice/status` | GET | STT/TTS availability; `enabled` false if `VOICE_ENABLED=false` |
| `/api/voice/voices` | GET | List available Kokoro voices |
| `/api/voice/transcribe` | POST | multipart `audio` field → `{"transcript": "...", "duration_ms": 123}` |
| `/api/voice/synthesise` | POST | `{"text", "voice", "speed"}` → WAV bytes |
---
## Modified Backend Files
### `src/fabledassistant/app.py`
- Register `voice_bp` blueprint
- In `startup()`: `asyncio.create_task(load_stt_model())` + `asyncio.create_task(load_tts_model())` when `VOICE_ENABLED`
### `src/fabledassistant/config.py`
- Add 4 new env var attributes
- Add validation in `validate()`
### `src/fabledassistant/services/llm.py`
- Add `voice_mode: bool = False` and `voice_speech_style: str = "conversational"` to `build_context()`
- When `voice_mode=True`, prepend: *"Respond naturally as if speaking aloud. No markdown, bullet points, headers, or code blocks. Complete sentences only."*
- Append style modifier based on `voice_speech_style`
### `src/fabledassistant/services/generation_task.py`
- Add `voice_mode: bool = False` to `run_generation()`
- Read `voice_speech_style` from settings when voice_mode; pass both to `build_context()`
### `src/fabledassistant/routes/chat.py`
- Allow `"voice"` in `conversation_type` whitelist
### `src/fabledassistant/services/chat.py`
- Exclude `conversation_type == "voice"` from auto-cleanup retention
---
## New Frontend Files
### `frontend/src/composables/useVoiceRecorder.ts`
Wraps `MediaRecorder`. Exports: `recording`, `error`, `isSupported`, `startRecording()`, `stopRecording() -> Promise<Blob>`.
### `frontend/src/composables/useVoiceAudio.ts`
Wraps `AudioContext`. Exports: `playing`, `isSupported`, `play(blob)`, `stop()`.
### `frontend/src/components/VoiceOverlay.vue`
Floating PTT button (fixed bottom-right). Creates/reuses a `"voice"` conversation. Full flow: record → transcribe → send → stream → synthesise → play. Space bar hotkey (from `App.vue`). Mounted globally in `App.vue`.
---
## Modified Frontend Files
### `frontend/src/api/client.ts`
Add: `transcribeAudio(blob)`, `synthesiseSpeech(text, voice?, speed?)`, `getVoiceStatus()`, `getVoiceList()`
### `frontend/src/views/BriefingView.vue`
- "Listen" button: reads latest assistant message aloud via TTS
- Mic button in input bar: PTT → transcribe → auto-fill input → send
- Auto-TTS on assistant response when in listen mode
### `frontend/src/views/SettingsView.vue`
- New "Voice" tab: voice dropdown, speed slider, speech style radio
- Loads from `/api/settings`, saves via `PUT /api/settings`
### `frontend/src/App.vue`
- Mount `<VoiceOverlay />`
- Space bar → `"voice:ptt-toggle"` custom event
---
## Audio Format
| Direction | Format | Rationale |
|---|---|---|
| Browser → Server | WebM/Opus | Native `MediaRecorder` output; no re-encoding |
| Server → Browser | WAV (24kHz, 16-bit mono) | Kokoro native; no re-encoding; `decodeAudioData` compatible |
---
## Dependencies to Add (`pyproject.toml`)
```toml
[project.optional-dependencies]
voice = [
"faster-whisper>=1.0",
"kokoro>=0.9",
"soundfile>=0.12",
]
```
Install unconditionally in Docker (activated by `VOICE_ENABLED` at runtime):
```dockerfile
RUN pip install faster-whisper kokoro soundfile
```
---
## Database Migration
No schema changes required. `conversation_type` is unconstrained TEXT. Voice settings use existing key-value `settings` table. Optional no-op migration `0034_voice_conversation_type.py` for audit trail.
---
## Implementation Phases
### Phase 1 — Backend services + routes
1. Add env vars to `config.py`
2. Create `services/stt.py` (faster-whisper)
3. Create `services/tts.py` (Kokoro)
4. Create `routes/voice.py` (4 endpoints)
5. Wire model loading into `app.py` startup
6. Add `voice_mode` to `build_context()` + `run_generation()`
7. Allow `"voice"` conversation type in chat route + cleanup exclusion
### Phase 2 — BriefingView listen + voice follow-up
1. Create `useVoiceRecorder.ts`
2. Create `useVoiceAudio.ts`
3. Add voice API functions to `client.ts`
4. Add "Listen" button + mic button to `BriefingView.vue`
### Phase 3 — VoiceOverlay for general voice chat
1. Create `VoiceOverlay.vue`
2. Mount in `App.vue` + Space bar hotkey
### Phase 4 — Settings UI
1. Add "Voice" tab to `SettingsView.vue`
---
## Kokoro Voice Reference
| ID | Character |
|---|---|
| `af_heart` | American female, warm (recommended default) |
| `af_bella` | American female, expressive |
| `af_nicole` | American female, breathy/intimate |
| `af_sarah` | American female, clear |
| `af_sky` | American female, bright |
| `am_adam` | American male, neutral |
| `am_michael` | American male, deeper |
| `bf_emma` | British female |
| `bf_isabella` | British female, formal |
| `bm_george` | British male |
| `bm_lewis` | British male, casual |
+2 -2
View File
@@ -298,7 +298,7 @@ async def fable_update_task(
@mcp.tool()
async def fable_add_task_log(task_id: int, body: str) -> dict:
async def fable_add_task_log(task_id: int, content: str) -> dict:
"""Append a timestamped progress log entry to a Fable task.
Use this to record work sessions, decisions, or status updates over time without
@@ -306,7 +306,7 @@ async def fable_add_task_log(task_id: int, body: str) -> dict:
chronologically in the task view.
"""
async with FableClient() as client:
return await tasks.add_task_log(client, task_id=task_id, body=body)
return await tasks.add_task_log(client, task_id=task_id, content=content)
# ---------------------------------------------------------------------------
+2 -2
View File
@@ -87,7 +87,7 @@ async def add_task_log(
client: FableClient,
*,
task_id: int,
body: str,
content: str,
) -> dict[str, Any]:
"""Append a log entry to a task."""
return await client.post(f"/api/tasks/{task_id}/logs", json={"body": body})
return await client.post(f"/api/tasks/{task_id}/logs", json={"content": content})
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "fable-mcp"
version = "0.1.0"
version = "0.2.0"
description = "MCP server for Fabled Assistant"
requires-python = ">=3.12"
dependencies = [
+3 -3
View File
@@ -46,9 +46,9 @@ async def test_update_task_patches(client):
@pytest.mark.asyncio
async def test_add_task_log_posts_to_logs_endpoint(client):
from fable_mcp.tools.tasks import add_task_log
client.post = AsyncMock(return_value={"id": 1, "body": "progress"})
await add_task_log(client, task_id=9, body="progress")
client.post = AsyncMock(return_value={"id": 1, "content": "progress"})
await add_task_log(client, task_id=9, content="progress")
client.post.assert_called_once()
path = client.post.call_args[0][0]
assert path == "/api/tasks/9/logs"
assert client.post.call_args[1]["json"]["body"] == "progress"
assert client.post.call_args[1]["json"]["content"] == "progress"
+29
View File
@@ -3,6 +3,7 @@ import { onMounted, onUnmounted, ref, watch } from "vue";
import { useRouter } from "vue-router";
import AppHeader from "@/components/AppHeader.vue";
import ToastNotification from "@/components/ToastNotification.vue";
import VoiceOverlay from "@/components/VoiceOverlay.vue";
import { useTheme } from "@/composables/useTheme";
import { useShortcuts } from "@/composables/useShortcuts";
import { useAuthStore } from "@/stores/auth";
@@ -47,9 +48,21 @@ function clearPrefix() {
prefixTimeout = null;
}
let spaceHeld = false;
function onGlobalKeydown(e: KeyboardEvent) {
if (!authStore.isAuthenticated) return;
// Space — PTT start (only when not typing, no modifiers, no repeat)
if (e.key === " " && !isInputActive() && !e.ctrlKey && !e.metaKey && !e.altKey && !e.repeat) {
e.preventDefault();
if (!spaceHeld) {
spaceHeld = true;
document.dispatchEvent(new CustomEvent("voice:ptt-toggle"));
}
return;
}
// ? — toggle shortcuts overlay (only when not typing)
if (e.key === "?" && !isInputActive() && !e.ctrlKey && !e.metaKey) {
toggleShortcuts();
@@ -122,8 +135,16 @@ function onGlobalKeydown(e: KeyboardEvent) {
}
}
function onGlobalKeyup(e: KeyboardEvent) {
if (e.key === " " && spaceHeld) {
spaceHeld = false;
document.dispatchEvent(new CustomEvent("voice:ptt-toggle"));
}
}
onMounted(async () => {
document.addEventListener("keydown", onGlobalKeydown);
document.addEventListener("keyup", onGlobalKeyup);
await authStore.checkAuth();
if (authStore.isAuthenticated) {
startAppServices();
@@ -149,6 +170,7 @@ watch(
onUnmounted(() => {
document.removeEventListener("keydown", onGlobalKeydown);
document.removeEventListener("keyup", onGlobalKeyup);
stopAppServices();
});
</script>
@@ -164,6 +186,9 @@ onUnmounted(() => {
<footer class="app-footer">v{{ appVersion }}</footer>
</div>
<!-- Global voice PTT overlay -->
<VoiceOverlay />
<!-- Keyboard shortcuts overlay -->
<Transition name="shortcuts-fade">
<div v-if="showShortcuts" class="shortcuts-overlay" @click.self="closeShortcuts">
@@ -268,6 +293,10 @@ onUnmounted(() => {
<kbd class="shortcut-key">Enter</kbd>
<span class="shortcut-desc">New line</span>
</div>
<div class="shortcut-row">
<kbd class="shortcut-key">Space</kbd>
<span class="shortcut-desc">Hold to speak (voice, when enabled)</span>
</div>
</div>
</div>
</div>
+48 -1
View File
@@ -123,7 +123,6 @@ export interface NotificationEntry {
export interface UserSearchResult {
id: number
username: string
email: string | null
}
// --- User search ---
@@ -620,3 +619,51 @@ export function getNewsItems(params: GetNewsItemsParams = {}) {
`/api/briefing/news?${p}`
)
}
// ─── Voice ────────────────────────────────────────────────────────────────────
export interface VoiceStatusResult {
enabled: boolean
stt: boolean
tts: boolean
stt_model?: string
tts_backend?: string
}
export interface VoiceEntry {
id: string
label: string
}
export const getVoiceStatus = () => apiGet<VoiceStatusResult>('/api/voice/status')
export const getVoiceList = () =>
apiGet<{ voices: VoiceEntry[] }>('/api/voice/voices').then(r => r.voices)
export async function transcribeAudio(blob: Blob): Promise<{ transcript: string; duration_ms: number }> {
const form = new FormData()
form.append('audio', blob, 'audio.webm')
const res = await fetch('/api/voice/transcribe', { method: 'POST', body: form })
if (!res.ok) {
const err = await res.json().catch(() => ({ error: `HTTP ${res.status}` }))
throw new ApiError(res.status, err)
}
return res.json()
}
export async function synthesiseSpeech(
text: string,
voice?: string,
speed?: number
): Promise<Blob> {
const res = await fetch('/api/voice/synthesise', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, voice: voice ?? 'af_heart', speed: speed ?? 1.0 }),
})
if (!res.ok) {
const err = await res.json().catch(() => ({ error: `HTTP ${res.status}` }))
throw new ApiError(res.status, err)
}
return res.blob()
}
-1
View File
@@ -138,7 +138,6 @@ onMounted(async () => {
<ul v-if="userResults.length" class="user-results">
<li v-for="u in userResults" :key="u.id" @click="selectUser(u)" class="user-result-item">
<span class="user-result-name">{{ u.username }}</span>
<span class="user-result-email">{{ u.email }}</span>
</li>
</ul>
</div>
+536
View File
@@ -0,0 +1,536 @@
<script setup lang="ts">
/**
* VoiceOverlay — global floating push-to-talk button.
*
* Full flow: record → transcribe → send to voice conv → stream → TTS → play.
* Manages its own "voice" conversation; does NOT touch the chat store so it
* never disrupts an open chat session.
*
* Space bar toggles PTT when no input field is focused (wired from App.vue via
* the "voice:ptt-toggle" custom event).
*/
import { ref, onMounted, onUnmounted, computed } from 'vue'
import { useVoiceRecorder } from '@/composables/useVoiceRecorder'
import { useVoiceAudio } from '@/composables/useVoiceAudio'
import { apiPost, apiSSEStream, getVoiceStatus, transcribeAudio, synthesiseSpeech } from '@/api/client'
// ─── Voice service availability ──────────────────────────────────────────────
const voiceEnabled = ref(false)
async function checkVoice() {
try {
const s = await getVoiceStatus()
voiceEnabled.value = s.enabled && s.stt && s.tts
} catch { /* feature absent */ }
}
// ─── Conversation management ─────────────────────────────────────────────────
const STORAGE_KEY = 'voice_overlay_conv_id'
const convId = ref<number | null>(Number(localStorage.getItem(STORAGE_KEY)) || null)
interface VoiceMessage { role: 'user' | 'assistant'; content: string }
const messages = ref<VoiceMessage[]>([])
async function ensureConversation(): Promise<number> {
if (convId.value) {
// Verify it still exists
try {
await fetch(`/api/chat/conversations/${convId.value}`)
.then((r) => { if (!r.ok) throw new Error('gone') })
return convId.value
} catch {
convId.value = null
localStorage.removeItem(STORAGE_KEY)
}
}
const conv = await apiPost<{ id: number }>('/api/chat/conversations', {
title: 'Voice',
conversation_type: 'voice',
})
convId.value = conv.id
localStorage.setItem(STORAGE_KEY, String(conv.id))
return conv.id
}
// ─── State machine ────────────────────────────────────────────────────────────
type Phase = 'idle' | 'recording' | 'transcribing' | 'generating' | 'speaking' | 'error'
const phase = ref<Phase>('idle')
const errorMsg = ref('')
const streamContent = ref('')
const open = ref(false)
const isBusy = computed(() => phase.value !== 'idle' && phase.value !== 'error')
// ─── Composables ─────────────────────────────────────────────────────────────
const recorder = useVoiceRecorder()
const audio = useVoiceAudio()
// ─── Core PTT flow ────────────────────────────────────────────────────────────
async function startPtt() {
if (!voiceEnabled.value || isBusy.value) return
errorMsg.value = ''
open.value = true
await recorder.startRecording()
if (recorder.error.value) {
phase.value = 'error'
errorMsg.value = recorder.error.value
return
}
phase.value = 'recording'
}
async function stopPtt() {
if (phase.value !== 'recording') return
phase.value = 'transcribing'
let blob: Blob
try {
blob = await recorder.stopRecording()
} catch {
phase.value = 'error'
errorMsg.value = 'Recording failed'
return
}
let transcript: string
try {
const result = await transcribeAudio(blob)
transcript = result.transcript.trim()
} catch {
phase.value = 'error'
errorMsg.value = 'Transcription failed'
return
}
if (!transcript) {
phase.value = 'idle'
return
}
messages.value.push({ role: 'user', content: transcript })
scrollToBottom()
// Send to voice conversation
phase.value = 'generating'
streamContent.value = ''
let cid: number
try {
cid = await ensureConversation()
} catch {
phase.value = 'error'
errorMsg.value = 'Could not create voice conversation'
return
}
let assistantMessageId: number
try {
const resp = await apiPost<{ assistant_message_id: number }>(
`/api/chat/conversations/${cid}/messages`,
{
content: transcript,
user_timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
}
)
assistantMessageId = resp.assistant_message_id
} catch {
phase.value = 'error'
errorMsg.value = 'Failed to send message'
return
}
// Stream the response
await new Promise<void>((resolve) => {
const handle = apiSSEStream(
`/api/chat/conversations/${cid}/generation/stream`,
(event) => {
switch (event.event) {
case 'chunk':
streamContent.value += event.data.chunk as string
break
case 'done':
handle.close()
resolve()
break
case 'error':
handle.close()
resolve()
break
}
}
)
// Safety timeout: 3 minutes
setTimeout(() => { handle.close(); resolve() }, 180_000)
})
const responseText = streamContent.value.trim()
if (responseText) {
messages.value.push({ role: 'assistant', content: responseText })
streamContent.value = ''
scrollToBottom()
}
// Synthesise and play
if (responseText) {
phase.value = 'speaking'
try {
const wavBlob = await synthesiseSpeech(responseText)
await audio.play(wavBlob)
} catch {
// TTS failure is non-critical; show response text
}
}
phase.value = 'idle'
assistantMessageId // consumed; suppress lint
}
function cancelAll() {
recorder.stopRecording().catch(() => {})
audio.stop()
phase.value = 'idle'
streamContent.value = ''
errorMsg.value = ''
}
// ─── Space bar PTT (event from App.vue) ──────────────────────────────────────
function onPttToggle() {
if (!voiceEnabled.value) return
if (phase.value === 'recording') {
stopPtt()
} else if (phase.value === 'idle' || phase.value === 'error') {
startPtt()
}
}
// ─── Scroll ───────────────────────────────────────────────────────────────────
const transcriptEl = ref<HTMLElement | null>(null)
function scrollToBottom() {
setTimeout(() => {
if (transcriptEl.value) {
transcriptEl.value.scrollTop = transcriptEl.value.scrollHeight
}
}, 50)
}
// ─── Lifecycle ────────────────────────────────────────────────────────────────
onMounted(() => {
checkVoice()
document.addEventListener('voice:ptt-toggle', onPttToggle)
})
onUnmounted(() => {
document.removeEventListener('voice:ptt-toggle', onPttToggle)
cancelAll()
})
</script>
<template>
<Teleport to="body">
<div v-if="voiceEnabled" class="voice-overlay">
<!-- Expanded transcript panel -->
<Transition name="panel-slide">
<div v-if="open && messages.length > 0" class="voice-panel">
<div class="voice-panel-header">
<span class="voice-panel-title">Voice</span>
<button class="voice-panel-close" @click="open = false" aria-label="Close">×</button>
</div>
<div class="voice-transcript" ref="transcriptEl">
<div
v-for="(msg, i) in messages.slice(-10)"
:key="i"
:class="['voice-msg', `voice-msg--${msg.role}`]"
>{{ msg.content }}</div>
<!-- Live streaming text -->
<div v-if="phase === 'generating' && streamContent" class="voice-msg voice-msg--assistant voice-msg--streaming">
{{ streamContent }}<span class="voice-cursor"></span>
</div>
</div>
</div>
</Transition>
<!-- PTT button -->
<div class="voice-btn-wrap">
<!-- Status label -->
<div v-if="phase !== 'idle'" class="voice-status-label">
<span v-if="phase === 'recording'">Recording</span>
<span v-else-if="phase === 'transcribing'">Transcribing</span>
<span v-else-if="phase === 'generating'">Thinking</span>
<span v-else-if="phase === 'speaking'">Speaking</span>
<span v-else-if="phase === 'error'" class="voice-error-label">{{ errorMsg || 'Error' }}</span>
</div>
<div v-else-if="!open || !messages.length" class="voice-status-label voice-hint">
Hold <kbd>Space</kbd> or tap
</div>
<!-- Cancel button (shown while busy or speaking) -->
<button
v-if="isBusy || audio.playing.value"
class="voice-cancel"
@click="cancelAll"
aria-label="Cancel"
title="Cancel"
>
<svg width="13" height="13" viewBox="0 0 24 24" fill="currentColor">
<path d="M19 6.41L17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12z"/>
</svg>
</button>
<!-- Main PTT button -->
<button
class="voice-ptt-btn"
:class="{
'voice-ptt--recording': phase === 'recording',
'voice-ptt--busy': phase === 'generating' || phase === 'transcribing',
'voice-ptt--speaking': phase === 'speaking' || audio.playing.value,
'voice-ptt--error': phase === 'error',
}"
@mousedown.prevent="startPtt"
@mouseup.prevent="stopPtt"
@touchstart.prevent="startPtt"
@touchend.prevent="stopPtt"
@click.prevent="phase === 'error' ? (phase = 'idle') : undefined"
:disabled="phase === 'transcribing' || phase === 'generating'"
:aria-label="phase === 'recording' ? 'Release to send' : 'Hold to speak'"
:title="phase === 'recording' ? 'Release to send' : 'Hold Space or tap to speak'"
>
<!-- Idle: mic icon -->
<svg v-if="phase === 'idle' || phase === 'error'" width="22" height="22" viewBox="0 0 24 24" fill="currentColor">
<path d="M12 14c1.66 0 3-1.34 3-3V5c0-1.66-1.34-3-3-3S9 3.34 9 5v6c0 1.66 1.34 3 3 3zm-1-9c0-.55.45-1 1-1s1 .45 1 1v6c0 .55-.45 1-1 1s-1-.45-1-1V5zm6 6c0 2.76-2.24 5-5 5s-5-2.24-5-5H5c0 3.53 2.61 6.43 6 6.92V21h2v-3.08c3.39-.49 6-3.39 6-6.92h-2z"/>
</svg>
<!-- Recording: waveform / stop icon -->
<svg v-else-if="phase === 'recording'" width="22" height="22" viewBox="0 0 24 24" fill="currentColor">
<path d="M6 19h4V5H6v14zm8-14v14h4V5h-4z"/>
</svg>
<!-- Busy: spinner dots -->
<span v-else-if="phase === 'transcribing' || phase === 'generating'" class="voice-spinner">
<span></span><span></span><span></span>
</span>
<!-- Speaking: sound waves -->
<svg v-else-if="phase === 'speaking'" width="22" height="22" viewBox="0 0 24 24" fill="currentColor">
<path d="M3 9v6h4l5 5V4L7 9H3zm13.5 3c0-1.77-1.02-3.29-2.5-4.03v8.05c1.48-.73 2.5-2.25 2.5-4.02zM14 3.23v2.06c2.89.86 5 3.54 5 6.71s-2.11 5.85-5 6.71v2.06c4.01-.91 7-4.49 7-8.77s-2.99-7.86-7-8.77z"/>
</svg>
</button>
</div>
</div>
</Teleport>
</template>
<style scoped>
.voice-overlay {
position: fixed;
bottom: 2rem;
right: 1.5rem;
z-index: 8000;
display: flex;
flex-direction: column;
align-items: flex-end;
gap: 0.5rem;
pointer-events: none;
}
/* ─── Panel ──────────────────────────────────────────────────────────────── */
.voice-panel {
pointer-events: all;
width: min(320px, 90vw);
background: var(--color-bg-card);
border: 1px solid var(--color-border);
border-radius: 14px;
box-shadow: 0 8px 32px var(--color-shadow, rgba(0,0,0,0.22));
overflow: hidden;
display: flex;
flex-direction: column;
max-height: 340px;
}
.voice-panel-header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 0.6rem 0.85rem 0.5rem;
border-bottom: 1px solid var(--color-border);
flex-shrink: 0;
}
.voice-panel-title {
font-size: 0.75rem;
font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.07em;
color: var(--color-primary);
}
.voice-panel-close {
background: none;
border: none;
font-size: 1.3rem;
line-height: 1;
color: var(--color-text-muted);
cursor: pointer;
padding: 0 0.1rem;
}
.voice-panel-close:hover { color: var(--color-text); }
.voice-transcript {
flex: 1;
overflow-y: auto;
padding: 0.65rem 0.85rem;
display: flex;
flex-direction: column;
gap: 0.45rem;
}
.voice-msg {
font-size: 0.875rem;
line-height: 1.45;
padding: 0.4rem 0.65rem;
border-radius: 10px;
max-width: 88%;
white-space: pre-wrap;
word-break: break-word;
}
.voice-msg--user {
align-self: flex-end;
background: color-mix(in srgb, var(--color-primary) 12%, transparent);
border: 1px solid color-mix(in srgb, var(--color-primary) 25%, transparent);
color: var(--color-text);
}
.voice-msg--assistant {
align-self: flex-start;
background: var(--color-bg-secondary);
border: 1px solid var(--color-border);
color: var(--color-text);
}
.voice-msg--streaming { opacity: 0.85; }
.voice-cursor {
display: inline-block;
animation: blink 0.9s step-end infinite;
margin-left: 1px;
color: var(--color-primary);
}
@keyframes blink { 0%, 100% { opacity: 1; } 50% { opacity: 0; } }
/* ─── Button cluster ─────────────────────────────────────────────────────── */
.voice-btn-wrap {
pointer-events: all;
display: flex;
flex-direction: column;
align-items: flex-end;
gap: 0.35rem;
}
.voice-status-label {
font-size: 0.72rem;
color: var(--color-text-muted);
background: var(--color-bg-card);
border: 1px solid var(--color-border);
border-radius: 6px;
padding: 0.18rem 0.5rem;
white-space: nowrap;
box-shadow: 0 2px 6px var(--color-shadow, rgba(0,0,0,0.12));
}
.voice-hint { opacity: 0.7; }
.voice-hint kbd {
font-family: ui-monospace, monospace;
font-size: 0.68rem;
padding: 0.05rem 0.25rem;
border: 1px solid var(--color-border);
border-radius: 3px;
background: var(--color-bg-secondary);
}
.voice-error-label { color: #ef4444; }
.voice-cancel {
background: var(--color-bg-card);
border: 1px solid var(--color-border);
border-radius: 50%;
width: 28px;
height: 28px;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
color: var(--color-text-muted);
box-shadow: 0 2px 6px var(--color-shadow, rgba(0,0,0,0.12));
transition: all 0.15s;
}
.voice-cancel:hover { color: #ef4444; border-color: #ef4444; }
.voice-ptt-btn {
width: 58px;
height: 58px;
border-radius: 50%;
border: none;
background: linear-gradient(135deg, #6366f1, #4f46e5);
color: #fff;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
box-shadow: 0 4px 16px rgba(99, 102, 241, 0.45);
transition: transform 0.12s, box-shadow 0.12s, background 0.2s;
touch-action: none;
user-select: none;
flex-shrink: 0;
}
.voice-ptt-btn:hover:not(:disabled) {
transform: scale(1.06);
box-shadow: 0 6px 20px rgba(99, 102, 241, 0.55);
}
.voice-ptt-btn:disabled { opacity: 0.6; cursor: not-allowed; }
.voice-ptt--recording {
background: linear-gradient(135deg, #ef4444, #dc2626) !important;
box-shadow: 0 4px 16px rgba(239, 68, 68, 0.5) !important;
animation: ptt-pulse 0.9s ease-in-out infinite;
}
.voice-ptt--busy {
background: linear-gradient(135deg, #8b5cf6, #7c3aed) !important;
box-shadow: 0 4px 16px rgba(139, 92, 246, 0.45) !important;
}
.voice-ptt--speaking {
background: linear-gradient(135deg, #10b981, #059669) !important;
box-shadow: 0 4px 16px rgba(16, 185, 129, 0.45) !important;
animation: ptt-pulse 1.4s ease-in-out infinite;
}
.voice-ptt--error {
background: linear-gradient(135deg, #6b7280, #4b5563) !important;
box-shadow: none !important;
}
@keyframes ptt-pulse {
0%, 100% { transform: scale(1); }
50% { transform: scale(1.08); }
}
/* ─── Spinner dots ───────────────────────────────────────────────────────── */
.voice-spinner {
display: flex;
gap: 4px;
align-items: center;
}
.voice-spinner span {
width: 5px;
height: 5px;
border-radius: 50%;
background: #fff;
animation: dot-bounce 1.2s ease-in-out infinite;
}
.voice-spinner span:nth-child(2) { animation-delay: 0.2s; }
.voice-spinner span:nth-child(3) { animation-delay: 0.4s; }
@keyframes dot-bounce {
0%, 80%, 100% { transform: scale(0.7); opacity: 0.5; }
40% { transform: scale(1); opacity: 1; }
}
/* ─── Transition ─────────────────────────────────────────────────────────── */
.panel-slide-enter-active,
.panel-slide-leave-active {
transition: opacity 0.2s ease, transform 0.2s ease;
}
.panel-slide-enter-from,
.panel-slide-leave-to {
opacity: 0;
transform: translateY(8px);
}
</style>
+60
View File
@@ -0,0 +1,60 @@
import { ref, readonly } from 'vue'
/**
* Audio playback composable wrapping the Web Audio API.
*
* Usage:
* const { playing, isSupported, play, stop } = useVoiceAudio()
* await play(wavBlob)
*/
export function useVoiceAudio() {
const playing = ref(false)
const isSupported = typeof AudioContext !== 'undefined' || typeof (window as unknown as Record<string, unknown>).webkitAudioContext !== 'undefined'
let audioCtx: AudioContext | null = null
let currentSource: AudioBufferSourceNode | null = null
function _getCtx(): AudioContext {
if (!audioCtx || audioCtx.state === 'closed') {
const Ctx = window.AudioContext ?? (window as unknown as Record<string, typeof AudioContext>).webkitAudioContext
audioCtx = new Ctx()
}
return audioCtx
}
async function play(blob: Blob): Promise<void> {
stop()
const ctx = _getCtx()
if (ctx.state === 'suspended') await ctx.resume()
const arrayBuffer = await blob.arrayBuffer()
const audioBuffer = await ctx.decodeAudioData(arrayBuffer)
const source = ctx.createBufferSource()
source.buffer = audioBuffer
source.connect(ctx.destination)
currentSource = source
playing.value = true
source.onended = () => {
playing.value = false
currentSource = null
}
source.start(0)
}
function stop(): void {
if (currentSource) {
try { currentSource.stop() } catch { /* already stopped */ }
currentSource = null
}
playing.value = false
}
return {
playing: readonly(playing),
isSupported,
play,
stop,
}
}
@@ -0,0 +1,92 @@
import { ref, readonly } from 'vue'
/**
* Push-to-talk recorder wrapping the browser MediaRecorder API.
*
* Usage:
* const { recording, error, isSupported, startRecording, stopRecording } = useVoiceRecorder()
* await startRecording()
* const blob = await stopRecording() // resolves with the recorded audio Blob
*/
export function useVoiceRecorder() {
const recording = ref(false)
const error = ref<string | null>(null)
const isSupported = typeof MediaRecorder !== 'undefined' && !!navigator.mediaDevices?.getUserMedia
let mediaRecorder: MediaRecorder | null = null
let chunks: Blob[] = []
let stream: MediaStream | null = null
let resolveStop: ((blob: Blob) => void) | null = null
let rejectStop: ((err: Error) => void) | null = null
async function startRecording(): Promise<void> {
error.value = null
if (!isSupported) {
error.value = 'Audio recording is not supported in this browser'
return
}
if (recording.value) return
try {
stream = await navigator.mediaDevices.getUserMedia({ audio: true })
} catch (e) {
error.value = 'Microphone access denied'
return
}
chunks = []
const mimeType = MediaRecorder.isTypeSupported('audio/webm;codecs=opus')
? 'audio/webm;codecs=opus'
: MediaRecorder.isTypeSupported('audio/webm')
? 'audio/webm'
: ''
mediaRecorder = mimeType ? new MediaRecorder(stream, { mimeType }) : new MediaRecorder(stream)
mediaRecorder.ondataavailable = (e) => {
if (e.data.size > 0) chunks.push(e.data)
}
mediaRecorder.onstop = () => {
const blob = new Blob(chunks, { type: mediaRecorder?.mimeType ?? 'audio/webm' })
chunks = []
stream?.getTracks().forEach((t) => t.stop())
stream = null
recording.value = false
resolveStop?.(blob)
resolveStop = null
rejectStop = null
}
mediaRecorder.onerror = () => {
recording.value = false
error.value = 'Recording error'
rejectStop?.(new Error('MediaRecorder error'))
resolveStop = null
rejectStop = null
}
mediaRecorder.start(100) // collect in 100ms chunks
recording.value = true
}
function stopRecording(): Promise<Blob> {
return new Promise((resolve, reject) => {
if (!mediaRecorder || !recording.value) {
reject(new Error('Not recording'))
return
}
resolveStop = resolve
rejectStop = reject
mediaRecorder.stop()
})
}
return {
recording: readonly(recording),
error: readonly(error),
isSupported,
startRecording,
stopRecording,
}
}
+1 -1
View File
@@ -38,7 +38,7 @@ const headingRenderer = {
marked.use({ renderer: headingRenderer });
const PURIFY_OPTS_FULL = {
ADD_ATTR: ["data-tag", "data-title", "src", "alt"],
ADD_ATTR: ["data-tag", "data-title"],
FORCE_BODY: true,
};
+200 -1
View File
@@ -1,6 +1,8 @@
<script setup lang="ts">
import { ref, computed, onMounted, watch, nextTick } from 'vue'
import { useBackgroundRefresh } from '@/composables/useBackgroundRefresh'
import { useVoiceRecorder } from '@/composables/useVoiceRecorder'
import { useVoiceAudio } from '@/composables/useVoiceAudio'
import { useChatStore } from '@/stores/chat'
import ChatMessage from '@/components/ChatMessage.vue'
import WeatherCard from '@/components/WeatherCard.vue'
@@ -15,6 +17,9 @@ import {
postRssReaction,
deleteRssReaction,
getNewsItems,
getVoiceStatus,
transcribeAudio,
synthesiseSpeech,
type BriefingConversation,
type BriefingMessage,
} from '@/api/client'
@@ -224,6 +229,87 @@ function convLabel(c: BriefingConversation): string {
return c.title || 'Briefing'
}
// ─── Voice ────────────────────────────────────────────────────────────────────
const voiceEnabled = ref(false)
const listenMode = ref(false)
const transcribing = ref(false)
const synthesising = ref(false)
const recorder = useVoiceRecorder()
const audio = useVoiceAudio()
// Check voice availability once on mount
async function checkVoice() {
try {
const status = await getVoiceStatus()
voiceEnabled.value = status.enabled && status.stt && status.tts
} catch { /* voice feature absent */ }
}
// Read the latest assistant message aloud
async function listenToLatest() {
const lastAssistant = [...messages.value].reverse().find((m) => m.role === 'assistant')
if (!lastAssistant?.content) return
await speakText(lastAssistant.content)
}
async function speakText(text: string) {
if (!voiceEnabled.value || synthesising.value) return
// Strip markdown for cleaner TTS output
const plain = text
.replace(/```[\s\S]*?```/g, '')
.replace(/`[^`]+`/g, (m) => m.slice(1, -1))
.replace(/#{1,6}\s+/g, '')
.replace(/\*\*([^*]+)\*\*/g, '$1')
.replace(/\*([^*]+)\*/g, '$1')
.replace(/\[([^\]]+)\]\([^)]+\)/g, '$1')
.replace(/^\s*[-*+]\s+/gm, '')
.replace(/\n{2,}/g, ' ')
.trim()
if (!plain) return
synthesising.value = true
try {
const blob = await synthesiseSpeech(plain)
await audio.play(blob)
} catch {
// TTS failure is non-critical
} finally {
synthesising.value = false
}
}
// PTT: hold to record, release to transcribe and send
async function startPtt() {
if (!voiceEnabled.value || recorder.recording.value) return
await recorder.startRecording()
}
async function stopPtt() {
if (!recorder.recording.value) return
transcribing.value = true
try {
const blob = await recorder.stopRecording()
const { transcript } = await transcribeAudio(blob)
if (transcript.trim()) {
input.value = transcript.trim()
await send()
}
} catch {
// transcription failure — leave input empty
} finally {
transcribing.value = false
}
}
// Auto-TTS when listen mode is on and streaming ends
watch(() => chatStore.streaming, async (streaming) => {
if (!streaming && listenMode.value && voiceEnabled.value) {
// Small delay to let messages update after stream
await new Promise((r) => setTimeout(r, 200))
await listenToLatest()
}
})
// Convert BriefingMessage to Message for ChatMessage component
function toMsg(m: BriefingMessage): Message {
return {
@@ -261,6 +347,7 @@ useBackgroundRefresh(
onMounted(async () => {
await checkSetup()
if (!showWizard.value) await loadAll()
checkVoice()
})
</script>
@@ -281,6 +368,29 @@ onMounted(async () => {
<select v-if="conversations.length" v-model="selectedConvId" class="briefing-conv-select">
<option v-for="c in conversations" :key="c.id" :value="c.id">{{ convLabel(c) }}</option>
</select>
<!-- Listen button: reads latest assistant message; toggles auto-TTS mode -->
<button
v-if="voiceEnabled"
class="btn-voice-header"
:class="{ 'btn-voice-active': listenMode, 'btn-voice-busy': synthesising || audio.playing.value }"
@click="listenMode ? (listenMode = false) : (listenMode = true, listenToLatest())"
:disabled="synthesising"
:title="listenMode ? 'Stop auto-read' : 'Listen to briefing'"
>
<svg v-if="!synthesising && !audio.playing.value" width="15" height="15" viewBox="0 0 24 24" fill="currentColor">
<path d="M3 9v6h4l5 5V4L7 9H3zm13.5 3c0-1.77-1.02-3.29-2.5-4.03v8.05c1.48-.73 2.5-2.25 2.5-4.02zM14 3.23v2.06c2.89.86 5 3.54 5 6.71s-2.11 5.85-5 6.71v2.06c4.01-.91 7-4.49 7-8.77s-2.99-7.86-7-8.77z"/>
</svg>
<svg v-else width="15" height="15" viewBox="0 0 24 24" fill="currentColor">
<path d="M12 3v10.55c-.59-.34-1.27-.55-2-.55-2.21 0-4 1.79-4 4s1.79 4 4 4 4-1.79 4-4V7h4V3h-6z"/>
</svg>
{{ listenMode ? 'Listening' : 'Listen' }}
</button>
<button
v-if="voiceEnabled && (synthesising || audio.playing.value)"
class="btn-trigger"
@click="audio.stop()"
title="Stop playback"
>Stop</button>
<button
class="btn-trigger"
@click="triggerNow"
@@ -343,10 +453,31 @@ onMounted(async () => {
<textarea
v-model="input"
class="briefing-input"
placeholder="Reply to your briefing…"
:placeholder="transcribing ? 'Transcribing…' : recorder.recording.value ? 'Recording…' : 'Reply to your briefing…'"
rows="1"
@keydown="onKeydown"
:disabled="transcribing || recorder.recording.value"
></textarea>
<!-- Mic PTT button (hold to record) -->
<button
v-if="voiceEnabled && recorder.isSupported"
class="btn-mic"
:class="{ 'btn-mic-active': recorder.recording.value, 'btn-mic-busy': transcribing }"
@mousedown.prevent="startPtt"
@mouseup.prevent="stopPtt"
@touchstart.prevent="startPtt"
@touchend.prevent="stopPtt"
:disabled="transcribing || chatStore.streaming || sending"
:title="recorder.recording.value ? 'Release to send' : 'Hold to speak'"
aria-label="Push to talk"
>
<svg v-if="!transcribing" width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
<path d="M12 14c1.66 0 3-1.34 3-3V5c0-1.66-1.34-3-3-3S9 3.34 9 5v6c0 1.66 1.34 3 3 3zm-1-9c0-.55.45-1 1-1s1 .45 1 1v6c0 .55-.45 1-1 1s-1-.45-1-1V5zm6 6c0 2.76-2.24 5-5 5s-5-2.24-5-5H5c0 3.53 2.61 6.43 6 6.92V21h2v-3.08c3.39-.49 6-3.39 6-6.92h-2z"/>
</svg>
<svg v-else width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
<path d="M12 4V1L8 5l4 4V6c3.31 0 6 2.69 6 6 0 1.01-.25 1.97-.7 2.8l1.46 1.46C19.54 15.03 20 13.57 20 12c0-4.42-3.58-8-8-8zm0 14c-3.31 0-6-2.69-6-6 0-1.01.25-1.97.7-2.8L5.24 7.74C4.46 8.97 4 10.43 4 12c0 4.42 3.58 8 8 8v3l4-4-4-4v3z"/>
</svg>
</button>
<button
class="btn-send"
@click="send"
@@ -706,6 +837,74 @@ a.news-title:hover { text-decoration: underline; color: var(--color-primary); }
background: color-mix(in srgb, var(--color-primary) 12%, transparent);
}
/* ─── Voice buttons ──────────────────────────────────────────────────────── */
.btn-voice-header {
display: flex;
align-items: center;
gap: 0.35rem;
padding: 0.35rem 0.75rem;
border: 1px solid var(--color-border);
border-radius: 6px;
background: var(--color-bg-card);
color: var(--color-text-muted);
font-size: 0.8rem;
cursor: pointer;
white-space: nowrap;
transition: all 0.15s;
font-family: inherit;
}
.btn-voice-header:hover:not(:disabled) {
border-color: var(--color-primary);
color: var(--color-primary);
}
.btn-voice-header:disabled { opacity: 0.5; cursor: not-allowed; }
.btn-voice-active {
border-color: var(--color-primary) !important;
color: var(--color-primary) !important;
background: color-mix(in srgb, var(--color-primary) 10%, transparent) !important;
}
.btn-voice-busy {
border-color: var(--color-primary);
color: var(--color-primary);
animation: pulse-border 1.2s ease-in-out infinite;
}
@keyframes pulse-border {
0%, 100% { opacity: 1; }
50% { opacity: 0.55; }
}
.btn-mic {
padding: 0.55rem 0.65rem;
background: var(--color-bg-secondary);
border: 1px solid var(--color-border);
border-radius: 10px;
color: var(--color-text-muted);
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
flex-shrink: 0;
transition: all 0.15s;
touch-action: none;
user-select: none;
}
.btn-mic:hover:not(:disabled) {
border-color: var(--color-primary);
color: var(--color-primary);
}
.btn-mic:disabled { opacity: 0.4; cursor: not-allowed; }
.btn-mic-active {
background: color-mix(in srgb, #ef4444 15%, transparent) !important;
border-color: #ef4444 !important;
color: #ef4444 !important;
animation: pulse-border 0.8s ease-in-out infinite;
}
.btn-mic-busy {
border-color: var(--color-primary);
color: var(--color-primary);
}
/* ─── Responsive ─────────────────────────────────────────────────────────── */
@media (max-width: 900px) {
+329 -4
View File
@@ -3,7 +3,7 @@ import { ref, watch, onMounted } from "vue";
import { useSettingsStore } from "@/stores/settings";
import { useAuthStore } from "@/stores/auth";
import { useToastStore } from "@/stores/toast";
import { apiGet, apiPost, apiPut, apiDelete, listGroups, createGroup, deleteGroup, listGroupMembers, addGroupMember, removeGroupMember, searchUsers, getBriefingConfig, saveBriefingConfig, getBriefingFeeds, createBriefingFeed, deleteBriefingFeed, refreshBriefingFeeds, geocodeAddress, getFableMcpInfo, listApiKeys, createApiKey as apiCreateApiKey, revokeApiKey as apiRevokeApiKey, type ApiKeyEntry, type GroupEntry, type GroupMember, type UserSearchResult, type BriefingConfig, type BriefingFeed } from "@/api/client";
import { apiGet, apiPost, apiPut, apiDelete, listGroups, createGroup, deleteGroup, listGroupMembers, addGroupMember, removeGroupMember, searchUsers, getBriefingConfig, saveBriefingConfig, getBriefingFeeds, createBriefingFeed, deleteBriefingFeed, refreshBriefingFeeds, geocodeAddress, getFableMcpInfo, listApiKeys, createApiKey as apiCreateApiKey, revokeApiKey as apiRevokeApiKey, getVoiceStatus, getVoiceList, type ApiKeyEntry, type GroupEntry, type GroupMember, type UserSearchResult, type BriefingConfig, type BriefingFeed, type VoiceStatusResult, type VoiceEntry } from "@/api/client";
import { usePushStore } from "@/stores/push";
import type { User } from "@/types/auth";
import PaginationBar from "@/components/PaginationBar.vue";
@@ -40,7 +40,7 @@ const appVersion = ref('dev');
const restoreFileInput = ref<HTMLInputElement | null>(null);
// Migrate stored "admin" → "config"; unknown tabs fall back to "general"
const VALID_TABS = new Set(["general", "account", "notifications", "integrations", "data", "briefing", "apikeys", "config", "users", "logs", "groups"]);
const VALID_TABS = new Set(["general", "account", "notifications", "integrations", "data", "briefing", "voice", "apikeys", "config", "users", "logs", "groups"]);
const _stored = localStorage.getItem("settings_tab") ?? "general";
const activeTab = ref(VALID_TABS.has(_stored) ? (_stored === "admin" ? "config" : _stored) : "general");
@@ -51,6 +51,7 @@ function _loadTabContent(tab: string) {
else if (tab === "groups") loadGroupsPanel();
}
if (tab === "briefing") loadBriefingTab();
if (tab === "voice") loadVoiceTab();
if (tab === "apikeys") { fetchApiKeys(); loadMcpInfo(); }
}
@@ -444,6 +445,56 @@ const baseUrl = ref("");
const savingBaseUrl = ref(false);
const baseUrlSaved = ref(false);
// Voice config (admin only)
const adminVoiceEnabled = ref(false);
const adminVoiceSttModel = ref("base.en");
const savingAdminVoice = ref(false);
const adminVoiceSaved = ref(false);
const voiceLoadingModels = ref(false);
async function saveAdminVoice() {
savingAdminVoice.value = true;
adminVoiceSaved.value = false;
try {
await apiPut("/api/admin/voice", {
voice_enabled: adminVoiceEnabled.value,
voice_stt_model: adminVoiceSttModel.value,
});
adminVoiceSaved.value = true;
setTimeout(() => { adminVoiceSaved.value = false; }, 2000);
voiceTabLoaded.value = false;
if (adminVoiceEnabled.value) {
await reloadVoiceModels();
}
} finally {
savingAdminVoice.value = false;
}
}
async function reloadVoiceModels() {
voiceLoadingModels.value = true;
try {
await apiPost("/api/admin/voice/reload", {});
// Poll /api/voice/status until both STT and TTS are ready (max 3 min)
const deadline = Date.now() + 3 * 60 * 1000;
while (Date.now() < deadline) {
await new Promise((r) => setTimeout(r, 2500));
try {
const status = await apiGet<{ enabled: boolean; stt: boolean; tts: boolean }>("/api/voice/status");
if (status.stt && status.tts) {
voiceStatus.value = status;
voiceTabLoaded.value = true;
break;
}
} catch { /* keep polling */ }
}
} catch {
toastStore.show("Failed to trigger model reload", "error");
} finally {
voiceLoadingModels.value = false;
}
}
// Search test (SearXNG)
const searxngConfigured = ref(false);
const searxngUrl = ref("");
@@ -452,6 +503,49 @@ const searchResults = ref<{ url: string; title: string; snippet: string }[]>([])
const searchLoading = ref(false);
const searchError = ref("");
// Voice settings
const voiceStatus = ref<VoiceStatusResult | null>(null);
const voiceStatusLoading = ref(false);
const availableVoices = ref<VoiceEntry[]>([]);
const voiceTtsVoice = ref("af_heart");
const voiceTtsSpeed = ref(1.0);
const voiceSpeechStyle = ref("conversational");
const savingVoice = ref(false);
const voiceSaved = ref(false);
const voiceTabLoaded = ref(false);
async function loadVoiceTab() {
if (voiceTabLoaded.value) return;
voiceTabLoaded.value = true;
voiceStatusLoading.value = true;
try {
voiceStatus.value = await getVoiceStatus();
if (voiceStatus.value.tts) {
availableVoices.value = await getVoiceList();
}
} catch {
// Voice feature may be disabled
} finally {
voiceStatusLoading.value = false;
}
}
async function saveVoiceSettings() {
savingVoice.value = true;
voiceSaved.value = false;
try {
await apiPut("/api/settings", {
voice_tts_voice: voiceTtsVoice.value,
voice_tts_speed: String(voiceTtsSpeed.value),
voice_speech_style: voiceSpeechStyle.value,
});
voiceSaved.value = true;
setTimeout(() => { voiceSaved.value = false; }, 2000);
} finally {
savingVoice.value = false;
}
}
onMounted(async () => {
try {
const v = await apiGet<{ version: string }>('/api/version')
@@ -485,6 +579,11 @@ onMounted(async () => {
notifySecurityAlerts.value = allSettings.notify_security_alerts !== "false";
}
// Load voice settings
if (allSettings.voice_tts_voice) voiceTtsVoice.value = allSettings.voice_tts_voice;
if (allSettings.voice_tts_speed) voiceTtsSpeed.value = parseFloat(allSettings.voice_tts_speed);
if (allSettings.voice_speech_style) voiceSpeechStyle.value = allSettings.voice_speech_style;
// Load CalDAV settings
try {
const caldavConfig = await apiGet<Record<string, string>>("/api/settings/caldav");
@@ -516,6 +615,13 @@ onMounted(async () => {
} catch {
// base URL not configured yet
}
try {
const vc = await apiGet<Record<string, string>>("/api/admin/voice");
adminVoiceEnabled.value = vc.voice_enabled === "true";
adminVoiceSttModel.value = vc.voice_stt_model ?? "base.en";
} catch {
// voice not configured yet
}
}
_loadTabContent(activeTab.value);
});
@@ -1134,7 +1240,7 @@ function formatUserDate(iso: string): string {
<div class="sidebar-group">
<div class="sidebar-group-label">User</div>
<button
v-for="tab in ['general', 'account', 'notifications', 'integrations', 'data', 'briefing', 'apikeys']"
v-for="tab in ['general', 'account', 'notifications', 'integrations', 'data', 'briefing', 'voice', 'apikeys']"
:key="tab"
:class="['sidebar-item', { active: activeTab === tab }]"
@click="activeTab = tab"
@@ -1810,6 +1916,104 @@ function formatUserDate(iso: string): string {
</div>
<!-- ── Voice ── -->
<div v-show="activeTab === 'voice'" class="settings-grid">
<section class="settings-section full-width">
<h2>Voice</h2>
<p class="section-desc">
Configure text-to-speech and speech-to-text for voice conversations.
Requires <code>VOICE_ENABLED=true</code> on the server.
</p>
<!-- Status indicator -->
<div v-if="voiceStatusLoading" class="field-hint">Loading voice status…</div>
<div v-else-if="voiceStatus === null" class="field-hint">Voice status unavailable.</div>
<div v-else class="voice-status-row">
<span :class="['status-badge', voiceStatus.enabled ? 'status-on' : 'status-off']">
{{ voiceStatus.enabled ? 'Enabled' : 'Disabled' }}
</span>
<span v-if="voiceStatus.enabled" class="status-badge" :class="voiceStatus.stt ? 'status-on' : 'status-off'">
STT {{ voiceStatus.stt ? 'ready' : 'loading' }}
</span>
<span v-if="voiceStatus.enabled" class="status-badge" :class="voiceStatus.tts ? 'status-on' : 'status-off'">
TTS {{ voiceStatus.tts ? 'ready' : 'loading' }}
</span>
<span v-if="voiceStatus.stt_model" class="field-hint" style="margin-left:0.5rem;">
Model: {{ voiceStatus.stt_model }}
</span>
</div>
<div v-if="!voiceStatus?.enabled" class="field-hint" style="margin-top:0.75rem;">
Voice is disabled. An administrator can enable it under
<strong>Admin → Config → Voice</strong>.
</div>
</section>
<section v-if="voiceStatus?.enabled" class="settings-section full-width">
<h2>Speech Style</h2>
<p class="section-desc">Controls how the assistant phrases spoken responses.</p>
<div class="field-group">
<label class="field-label">Style</label>
<div class="radio-group">
<label v-for="opt in [
{ value: 'conversational', label: 'Conversational', hint: 'Warm and natural, like speaking to a friend' },
{ value: 'concise', label: 'Concise', hint: 'Short and to the point' },
{ value: 'detailed', label: 'Detailed', hint: 'Thorough explanations spoken aloud' },
]" :key="opt.value" class="radio-option">
<input type="radio" v-model="voiceSpeechStyle" :value="opt.value" />
<span>
<strong>{{ opt.label }}</strong>
<span class="field-hint">{{ opt.hint }}</span>
</span>
</label>
</div>
</div>
</section>
<section v-if="voiceStatus?.enabled" class="settings-section full-width">
<h2>Voice &amp; Speed</h2>
<p class="section-desc">Choose the TTS voice and playback speed.</p>
<div class="field-group">
<label class="field-label" for="voice-select">Voice</label>
<select id="voice-select" class="input" v-model="voiceTtsVoice" :disabled="availableVoices.length === 0">
<option v-if="availableVoices.length === 0" value="af_heart">af_heart (default)</option>
<option v-for="v in availableVoices" :key="v.id" :value="v.id">{{ v.label }}</option>
</select>
<p class="field-hint">Voice character and accent. Applies to all voice conversations.</p>
</div>
<div class="field-group">
<label class="field-label" for="voice-speed">
Speed: {{ voiceTtsSpeed.toFixed(1) }}×
</label>
<input
id="voice-speed"
type="range"
min="0.7"
max="1.3"
step="0.05"
v-model.number="voiceTtsSpeed"
class="range-input"
/>
<div class="range-labels">
<span>0.7× (slow)</span>
<span>1.0× (normal)</span>
<span>1.3× (fast)</span>
</div>
</div>
<div class="form-actions">
<button class="btn-save" @click="saveVoiceSettings" :disabled="savingVoice">
{{ voiceSaved ? 'Saved ' : savingVoice ? 'Saving' : 'Save Voice Settings' }}
</button>
</div>
</section>
</div>
<!-- ── API Keys ── -->
<div v-show="activeTab === 'apikeys'" class="settings-grid">
<section class="settings-section full-width">
@@ -1998,6 +2202,50 @@ FABLE_API_KEY=&lt;your-api-key&gt;</pre>
</div>
</section>
<section class="settings-section full-width">
<h2>Voice (Speech-to-Speech)</h2>
<p class="section-desc">
Enable self-hosted voice using faster-whisper (STT) and Kokoro (TTS).
Models load at startup — a server restart is required after changing these settings.
Install dependencies first: <code>pip install faster-whisper kokoro soundfile</code>
</p>
<div class="checkbox-field">
<label>
<input type="checkbox" v-model="adminVoiceEnabled" />
Enable Voice
</label>
<p class="field-hint">Shows the PTT button and voice settings to all users.</p>
</div>
<div v-if="adminVoiceEnabled" class="field" style="max-width:280px; margin-top:0.75rem;">
<label for="stt-model">STT Model</label>
<select id="stt-model" v-model="adminVoiceSttModel" class="input">
<option value="tiny.en">tiny.en — fastest, lowest accuracy (~75 MB)</option>
<option value="base.en">base.en — balanced (recommended, ~145 MB)</option>
<option value="small.en">small.en — better accuracy (~465 MB)</option>
<option value="medium.en">medium.en — best accuracy (~1.5 GB)</option>
</select>
<p class="field-hint">Larger models are more accurate but use more RAM and take longer to load.</p>
</div>
<div class="actions" style="margin-top:0.85rem;">
<button class="btn-save" @click="saveAdminVoice" :disabled="savingAdminVoice || voiceLoadingModels">
{{ savingAdminVoice ? 'Saving' : adminVoiceSaved ? 'Saved ' : 'Save' }}
</button>
<button
v-if="!voiceLoadingModels && adminVoiceEnabled"
class="btn-secondary"
@click="reloadVoiceModels"
:disabled="voiceLoadingModels"
title="Reload models without restarting the server"
>Reload models</button>
<span v-if="voiceLoadingModels" class="field-hint">
Loading models… this may take a minute.
<span class="voice-admin-spinner">
<span></span><span></span><span></span>
</span>
</span>
</div>
</section>
</div>
<!-- ── Users ── -->
@@ -2263,7 +2511,6 @@ FABLE_API_KEY=&lt;your-api-key&gt;</pre>
<ul v-if="groupMemberResults.length" class="member-results">
<li v-for="u in groupMemberResults" :key="u.id" class="member-result-item" @click="addMemberToGroup(g.id, u)">
<span class="member-result-name">{{ u.username }}</span>
<span class="member-result-email">{{ u.email }}</span>
</li>
</ul>
</div>
@@ -3529,4 +3776,82 @@ FABLE_API_KEY=&lt;your-api-key&gt;</pre>
word-break: break-all;
overflow-x: auto;
}
/* Voice tab */
.voice-status-row {
display: flex;
align-items: center;
gap: 0.5rem;
flex-wrap: wrap;
margin-top: 0.5rem;
}
.status-badge {
font-size: 0.75rem;
font-weight: 600;
padding: 0.15rem 0.55rem;
border-radius: 999px;
}
.status-on {
background: color-mix(in srgb, var(--color-success, #22c55e) 15%, transparent);
color: var(--color-success, #22c55e);
border: 1px solid color-mix(in srgb, var(--color-success, #22c55e) 40%, transparent);
}
.status-off {
background: color-mix(in srgb, var(--color-text-muted) 10%, transparent);
color: var(--color-text-muted);
border: 1px solid var(--color-border);
}
.radio-group {
display: flex;
flex-direction: column;
gap: 0.65rem;
margin-top: 0.35rem;
}
.radio-option {
display: flex;
align-items: flex-start;
gap: 0.55rem;
cursor: pointer;
}
.radio-option input[type="radio"] { margin-top: 0.2rem; flex-shrink: 0; }
.radio-option span { display: flex; flex-direction: column; gap: 0.15rem; }
.radio-option strong { font-size: 0.875rem; }
.range-input {
width: 100%;
max-width: 360px;
accent-color: var(--color-primary);
margin: 0.35rem 0 0.2rem;
}
.range-labels {
display: flex;
justify-content: space-between;
max-width: 360px;
font-size: 0.75rem;
color: var(--color-text-muted);
}
.form-actions {
margin-top: 1rem;
}
.voice-admin-spinner {
display: inline-flex;
gap: 3px;
align-items: center;
margin-left: 0.4rem;
vertical-align: middle;
}
.voice-admin-spinner span {
display: inline-block;
width: 4px;
height: 4px;
border-radius: 50%;
background: var(--color-text-muted);
animation: va-dot-bounce 1.2s ease-in-out infinite;
}
.voice-admin-spinner span:nth-child(2) { animation-delay: 0.2s; }
.voice-admin-spinner span:nth-child(3) { animation-delay: 0.4s; }
@keyframes va-dot-bounce {
0%, 80%, 100% { transform: scale(0.6); opacity: 0.4; }
40% { transform: scale(1); opacity: 1; }
}
</style>
+5
View File
@@ -29,6 +29,11 @@ dev = [
"pytest-asyncio>=0.23",
"ruff>=0.6",
]
voice = [
"faster-whisper>=1.0",
"kokoro>=0.9",
"soundfile>=0.12",
]
[tool.setuptools.packages.find]
where = ["src"]
+15
View File
@@ -31,6 +31,7 @@ from fabledassistant.routes.users import users_bp
from fabledassistant.routes.api_keys import api_keys_bp
from fabledassistant.routes.events import events_bp
from fabledassistant.routes.search import search_bp
from fabledassistant.routes.voice import voice_bp
STATIC_DIR = Path(__file__).parent / "static"
logger = logging.getLogger(__name__)
@@ -62,6 +63,13 @@ def create_app() -> Quart:
"Set SECRET_KEY or SECRET_KEY_FILE for production use."
)
if not Config.TRUST_PROXY_HEADERS:
logger.warning(
"TRUST_PROXY_HEADERS is not set. If this instance is behind a reverse proxy "
"(nginx, Caddy, Traefik) set TRUST_PROXY_HEADERS=true so rate limiting uses "
"real client IPs rather than the proxy IP."
)
app.register_blueprint(admin_bp)
app.register_blueprint(api)
app.register_blueprint(auth_bp)
@@ -85,6 +93,7 @@ def create_app() -> Quart:
app.register_blueprint(api_keys_bp)
app.register_blueprint(events_bp)
app.register_blueprint(search_bp)
app.register_blueprint(voice_bp)
@app.before_request
async def before_request():
@@ -257,6 +266,12 @@ def create_app() -> Quart:
from fabledassistant.services.briefing_scheduler import start_briefing_scheduler
await start_briefing_scheduler(asyncio.get_running_loop())
# Voice model loading (enabled via Admin → Config in the UI, or VOICE_ENABLED env var)
from fabledassistant.services.stt import load_stt_model
from fabledassistant.services.tts import load_tts_model
asyncio.create_task(load_stt_model())
asyncio.create_task(load_tts_model())
@app.after_serving
async def shutdown():
from fabledassistant.services.briefing_scheduler import stop_briefing_scheduler
+17
View File
@@ -66,6 +66,12 @@ class Config:
VAPID_PUBLIC_KEY: str = os.environ.get("VAPID_PUBLIC_KEY", "")
VAPID_CLAIMS_SUB: str = os.environ.get("VAPID_CLAIMS_SUB", "mailto:admin@fabledassistant.local")
# Voice (Speech-to-Speech) feature
VOICE_ENABLED: bool = os.environ.get("VOICE_ENABLED", "").lower() in ("1", "true", "yes")
STT_BACKEND: str = os.environ.get("STT_BACKEND", "faster-whisper")
STT_MODEL: str = os.environ.get("STT_MODEL", "base.en")
TTS_BACKEND: str = os.environ.get("TTS_BACKEND", "kokoro")
@classmethod
def oidc_enabled(cls) -> bool:
return bool(cls.OIDC_ISSUER and cls.OIDC_CLIENT_ID and cls.OIDC_CLIENT_SECRET)
@@ -88,5 +94,16 @@ class Config:
errors.append(f"SMTP_PORT={cls.SMTP_PORT} must be between 1 and 65535")
if cls.oidc_enabled() and not cls.BASE_URL.startswith(("http://", "https://")):
errors.append(f"BASE_URL='{cls.BASE_URL}' must start with http:// or https:// when OIDC is enabled")
if cls.SECRET_KEY == "dev-secret-change-me" and cls.SECURE_COOKIES:
errors.append(
"SECRET_KEY is set to the insecure default but SECURE_COOKIES=true indicates "
"a production deployment. Set SECRET_KEY or SECRET_KEY_FILE before starting."
)
_valid_stt_models = {"tiny.en", "base.en", "small.en", "medium.en"}
if cls.VOICE_ENABLED and cls.STT_MODEL not in _valid_stt_models:
errors.append(
f"STT_MODEL='{cls.STT_MODEL}' is not supported. "
f"Valid values: {', '.join(sorted(_valid_stt_models))}"
)
if errors:
raise ValueError("Configuration errors:\n" + "\n".join(f" - {e}" for e in errors))
+9
View File
@@ -1,3 +1,12 @@
"""In-process sliding-window rate limiter.
IMPORTANT — deployment note:
Rate limit counters are stored in memory and are lost on process restart.
When deployed behind a reverse proxy (nginx, Caddy, Traefik) you MUST set
TRUST_PROXY_HEADERS=true so that the real client IP is used as the bucket key
rather than the proxy's IP (which would cause all users to share one bucket).
"""
import asyncio
import time
from collections import defaultdict
+39
View File
@@ -19,6 +19,7 @@ from fabledassistant.services.backup import (
restore_full_backup,
)
from fabledassistant.services.email import SMTP_SETTING_KEYS, get_base_url, get_smtp_config, is_smtp_configured, send_test_email
from fabledassistant.services.voice_config import get_voice_config
from fabledassistant.services.logging import get_logs, get_log_stats, log_audit
from fabledassistant.services.notifications import send_invitation_email
from fabledassistant.services.settings import set_setting, set_settings_batch
@@ -200,6 +201,44 @@ async def update_base_url():
return jsonify({"status": "ok"})
@admin_bp.route("/voice", methods=["GET"])
@admin_required
async def get_voice_config_route():
config = await get_voice_config()
return jsonify(config)
@admin_bp.route("/voice", methods=["PUT"])
@admin_required
async def update_voice_config():
data = await request.get_json()
uid = get_current_user_id()
valid_models = {"tiny.en", "base.en", "small.en", "medium.en"}
settings: dict[str, str] = {}
if "voice_enabled" in data:
settings["voice_enabled"] = "true" if data["voice_enabled"] else "false"
if "voice_stt_model" in data:
model = str(data["voice_stt_model"])
if model not in valid_models:
return jsonify({"error": f"Invalid STT model. Choose from: {', '.join(sorted(valid_models))}"}), 400
settings["voice_stt_model"] = model
if settings:
await set_settings_batch(uid, settings)
await log_audit("voice_config", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details=settings)
return jsonify({"status": "ok"})
@admin_bp.route("/voice/reload", methods=["POST"])
@admin_required
async def reload_voice_models():
"""Reload STT and TTS models in the background without a server restart."""
from fabledassistant.services.stt import reload_stt_model
from fabledassistant.services.tts import reload_tts_model
asyncio.create_task(reload_stt_model())
asyncio.create_task(reload_tts_model())
return jsonify({"status": "loading"})
@admin_bp.route("/invitations", methods=["POST"])
@admin_required
async def create_invite():
+4 -1
View File
@@ -359,7 +359,10 @@ async def oauth_callback():
return redirect("/login?error=oauth")
sub = claims.get("sub", "")
email = claims.get("email", "")
# Only trust the email claim for account linking if the provider has verified it.
# An unverified email could be used to hijack an existing local account.
email_verified = claims.get("email_verified", False)
email = claims.get("email", "") if email_verified else ""
preferred_username = claims.get("preferred_username", "")
if not sub:
+6
View File
@@ -71,6 +71,12 @@ async def add_feed():
url = (data.get("url") or "").strip()
if not url:
return jsonify({"error": "url required"}), 400
scheme = url.split("://")[0].lower() if "://" in url else ""
if scheme not in ("http", "https"):
return jsonify({"error": "Feed URL must use http or https"}), 400
from fabledassistant.services.llm import _is_private_url
if _is_private_url(url):
return jsonify({"error": "Feed URL must not point to an internal address"}), 400
category = data.get("category") or None
async with async_session() as session:
+5 -4
View File
@@ -5,7 +5,7 @@ import logging
import httpx
from quart import Blueprint, Response, jsonify, request
from fabledassistant.auth import login_required, get_current_user_id
from fabledassistant.auth import admin_required, login_required, get_current_user_id
from fabledassistant.routes.utils import not_found, parse_pagination
from fabledassistant.config import Config
from fabledassistant.services.chat import (
@@ -76,7 +76,7 @@ async def create_conversation_route():
model = data.get("model", Config.OLLAMA_MODEL)
conversation_type = data.get("conversation_type", "chat")
# Only allow known types to prevent accidental misuse
if conversation_type not in ("chat", "mcp"):
if conversation_type not in ("chat", "mcp", "voice"):
conversation_type = "chat"
conv = await create_conversation(uid, title=title, model=model, conversation_type=conversation_type)
return jsonify(conv.to_dict()), 201
@@ -187,6 +187,7 @@ async def send_message_route(conv_id: int):
rag_project_id=effective_rag_project_id,
workspace_project_id=workspace_project_id,
user_timezone=user_timezone,
voice_mode=(conv.conversation_type == "voice"),
))
return jsonify({
@@ -439,7 +440,7 @@ async def list_models_route():
@chat_bp.route("/models/pull", methods=["POST"])
@login_required
@admin_required
async def pull_model_route():
"""Pull a model from Ollama, streaming progress via SSE."""
data = await request.get_json()
@@ -478,7 +479,7 @@ async def pull_model_route():
@chat_bp.route("/models/delete", methods=["POST"])
@login_required
@admin_required
async def delete_model_route():
"""Delete a model from Ollama."""
data = await request.get_json()
+3 -6
View File
@@ -1,14 +1,10 @@
"""Serve locally-cached images.
No authentication required — image IDs are opaque and unguessable (based
on the SHA-256 of the original URL). Images are served with a 1-day
Cache-Control header so the browser doesn't re-request on every page load.
"""
"""Serve locally-cached images."""
import logging
from quart import Blueprint, jsonify, send_file
from fabledassistant.auth import login_required
from fabledassistant.services.images import get_image_path, get_image_record
logger = logging.getLogger(__name__)
@@ -17,6 +13,7 @@ images_bp = Blueprint("images", __name__, url_prefix="/api/images")
@images_bp.route("/<int:image_id>", methods=["GET"])
@login_required
async def serve_image(image_id: int):
"""Serve a locally-cached image by its DB ID."""
record = await get_image_record(image_id)
+4
View File
@@ -50,12 +50,16 @@ async def create_milestone_route(project_id: int):
data = await request.get_json()
if not data.get("title"):
return jsonify({"error": "title is required"}), 400
status = data.get("status", "active")
if status not in ("active", "done"):
return jsonify({"error": "status must be 'active' or 'done'"}), 400
milestone = await create_milestone(
uid,
project_id,
title=data["title"],
description=data.get("description"),
order_index=data.get("order_index", 0),
status=status,
)
return jsonify(await _milestone_dict(milestone)), 201
+4
View File
@@ -38,12 +38,16 @@ async def create_project_route():
data = await request.get_json()
if not data.get("title"):
return jsonify({"error": "title is required"}), 400
status = data.get("status", "active")
if status not in ("active", "archived"):
return jsonify({"error": "status must be 'active' or 'archived'"}), 400
project = await create_project(
uid,
title=data["title"],
description=data.get("description", ""),
goal=data.get("goal", ""),
color=data.get("color"),
status=status,
)
return jsonify(project.to_dict()), 201
+11 -1
View File
@@ -3,7 +3,7 @@ from quart import Blueprint, jsonify, request
from fabledassistant.auth import login_required, get_current_user_id
from fabledassistant.config import Config
from fabledassistant.services.caldav import CALDAV_SETTING_KEYS, get_caldav_config, test_connection
from fabledassistant.services.llm import get_installed_models
from fabledassistant.services.llm import get_installed_models, _is_private_url
from fabledassistant.services.settings import delete_setting, get_all_settings, set_settings_batch
settings_bp = Blueprint("settings", __name__, url_prefix="/api/settings")
@@ -77,6 +77,16 @@ async def update_caldav():
uid = get_current_user_id()
data = await request.get_json()
# Validate CalDAV URL before saving — block internal/private addresses
if "caldav_url" in data:
url = str(data.get("caldav_url") or "").strip()
if url:
parsed_scheme = url.split("://")[0].lower() if "://" in url else ""
if parsed_scheme not in ("http", "https"):
return jsonify({"error": "CalDAV URL must use http or https"}), 400
if _is_private_url(url):
return jsonify({"error": "CalDAV URL must not point to an internal or private address"}), 400
settings_to_save = {}
for key in CALDAV_SETTING_KEYS:
if key in data:
+1 -1
View File
@@ -24,6 +24,6 @@ async def search_users():
).limit(10)
)).scalars().all()
return jsonify({"users": [
{"id": u.id, "username": u.username, "email": u.email}
{"id": u.id, "username": u.username}
for u in users
]})
+136
View File
@@ -0,0 +1,136 @@
"""Voice (Speech-to-Speech) routes at /api/voice."""
import logging
import time
from quart import Blueprint, jsonify, request
from fabledassistant.auth import login_required
logger = logging.getLogger(__name__)
voice_bp = Blueprint("voice", __name__, url_prefix="/api/voice")
@voice_bp.route("/status", methods=["GET"])
@login_required
async def voice_status():
"""Return availability of STT and TTS services."""
from fabledassistant.services.voice_config import get_voice_config
from fabledassistant.services.stt import stt_available
from fabledassistant.services.tts import tts_available
config = await get_voice_config()
enabled = config.get("voice_enabled", "false").lower() in ("1", "true", "yes")
if not enabled:
return jsonify({"enabled": False, "stt": False, "tts": False})
return jsonify({
"enabled": True,
"stt": stt_available(),
"tts": tts_available(),
"stt_model": config.get("voice_stt_model", "base.en"),
"tts_backend": "kokoro",
})
@voice_bp.route("/voices", methods=["GET"])
@login_required
async def list_voices():
"""Return available Kokoro voice IDs and labels."""
from fabledassistant.services.voice_config import is_voice_enabled
if not await is_voice_enabled():
return jsonify({"error": "Voice feature is disabled"}), 503
from fabledassistant.services.tts import list_voices, tts_available
if not tts_available():
return jsonify({"error": "TTS not available"}), 503
return jsonify({"voices": list_voices()})
@voice_bp.route("/transcribe", methods=["POST"])
@login_required
async def transcribe_audio():
"""Accept a multipart audio file and return the transcript.
Request: multipart/form-data with field 'audio' (WebM/Opus blob)
Response: {"transcript": "...", "duration_ms": 123}
"""
from fabledassistant.services.voice_config import is_voice_enabled
if not await is_voice_enabled():
return jsonify({"error": "Voice feature is disabled"}), 503
from fabledassistant.services.stt import stt_available, transcribe
if not stt_available():
return jsonify({"error": "STT not available — model may still be loading"}), 503
files = await request.files
audio_file = files.get("audio")
if audio_file is None:
return jsonify({"error": "No audio file provided"}), 400
audio_bytes = audio_file.read()
if not audio_bytes:
return jsonify({"error": "Empty audio file"}), 400
if len(audio_bytes) > 25 * 1024 * 1024: # 25 MB hard cap
return jsonify({"error": "Audio file too large (max 25 MB)"}), 413
mime_type = audio_file.content_type or "audio/webm"
t0 = time.monotonic()
try:
transcript = await transcribe(audio_bytes, mime_type)
except Exception:
logger.exception("STT transcription failed")
return jsonify({"error": "Transcription failed"}), 500
duration_ms = round((time.monotonic() - t0) * 1000)
return jsonify({"transcript": transcript, "duration_ms": duration_ms})
@voice_bp.route("/synthesise", methods=["POST"])
@login_required
async def synthesise_speech():
"""Convert text to speech and return WAV bytes.
Request body: {"text": "...", "voice": "af_heart", "speed": 1.0}
Response: audio/wav bytes
"""
from fabledassistant.services.voice_config import is_voice_enabled
if not await is_voice_enabled():
return jsonify({"error": "Voice feature is disabled"}), 503
from fabledassistant.services.tts import synthesise, tts_available
if not tts_available():
return jsonify({"error": "TTS not available — model may still be loading"}), 503
data = await request.get_json()
if not data:
return jsonify({"error": "JSON body required"}), 400
text = str(data.get("text", "")).strip()
if not text:
return jsonify({"error": "text is required"}), 400
if len(text) > 8000:
return jsonify({"error": "text too long (max 8000 characters)"}), 400
voice = str(data.get("voice", "af_heart"))
try:
speed = float(data.get("speed", 1.0))
except (TypeError, ValueError):
speed = 1.0
try:
wav_bytes = await synthesise(text, voice=voice, speed=speed)
except Exception:
logger.exception("TTS synthesis failed")
return jsonify({"error": "Synthesis failed"}), 500
from quart import Response
return Response(wav_bytes, mimetype="audio/wav")
+2 -1
View File
@@ -131,7 +131,8 @@ async def cleanup_old_conversations(user_id: int, days: int) -> int:
.where(
Conversation.user_id == user_id,
Conversation.updated_at < cutoff,
Conversation.conversation_type != "mcp", # preserve MCP audit trail
Conversation.conversation_type != "mcp", # preserve MCP audit trail
Conversation.conversation_type != "voice", # voice convs managed separately
)
.returning(Conversation.id)
)
@@ -152,6 +152,7 @@ async def run_generation(
rag_project_id: int | None = None,
workspace_project_id: int | None = None,
user_timezone: str | None = None,
voice_mode: bool = False,
) -> None:
"""Stream LLM response into buffer with periodic DB flushes."""
MAX_TOOL_ROUNDS = 5
@@ -177,6 +178,12 @@ async def run_generation(
# Phase 3: Build context and wait for model in parallel.
model_load_task = asyncio.create_task(wait_for_model_loaded(model, timeout=180.0))
# Fetch voice_speech_style from user settings when voice_mode is active.
voice_speech_style = "conversational"
if voice_mode:
from fabledassistant.services.settings import get_setting
voice_speech_style = await get_setting(user_id, "voice_speech_style", "conversational")
context_task = asyncio.create_task(build_context(
user_id, history_to_use, context_note_id, user_content,
history_summary=history_summary,
@@ -186,6 +193,8 @@ async def run_generation(
workspace_project_id=workspace_project_id,
user_timezone=user_timezone,
conv_id=conv_id,
voice_mode=voice_mode,
voice_speech_style=voice_speech_style,
))
messages, context_meta = await context_task
+27
View File
@@ -7,7 +7,9 @@ twice shares one file and one DB row.
"""
import hashlib
import ipaddress
import logging
import socket
from pathlib import Path
from urllib.parse import urlparse
@@ -21,6 +23,27 @@ from fabledassistant.models.image_cache import ImageCache
logger = logging.getLogger(__name__)
def _is_safe_image_url(url: str) -> bool:
"""Return True only if the URL is a public http/https address (SSRF guard)."""
try:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
return False
host = parsed.hostname
if not host:
return False
if host.lower() in ("localhost", "::1"):
return False
addr_info = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
for entry in addr_info:
ip = ipaddress.ip_address(entry[4][0])
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_reserved or ip.is_multicast:
return False
return True
except Exception:
return False # Block on resolution failure
_ALLOWED_TYPES = {
"image/jpeg",
"image/png",
@@ -59,6 +82,10 @@ async def fetch_and_store_image(
Returns None if the URL is unreachable, not a valid image type, or
exceeds IMAGE_MAX_BYTES.
"""
if not _is_safe_image_url(url):
logger.warning("Blocked image fetch of private/internal URL: %s", url[:80])
return None
url_hash = _url_hash(url)
# Return existing record if already cached
+15
View File
@@ -454,6 +454,8 @@ async def build_context(
workspace_project_id: int | None = None,
user_timezone: str | None = None,
conv_id: int | None = None,
voice_mode: bool = False,
voice_speech_style: str = "conversational",
) -> tuple[list[dict], dict]:
"""Build messages array for Ollama with system prompt and context.
@@ -516,6 +518,19 @@ async def build_context(
f"{tool_guidance}"
]
if voice_mode:
_style_hints = {
"conversational": "Be warm, natural, and conversational — like speaking to a friend.",
"concise": "Be brief and to the point. One or two sentences maximum unless detail is essential.",
"detailed": "Give thorough, informative responses as if narrating an explanation aloud.",
}
style_hint = _style_hints.get(voice_speech_style, _style_hints["conversational"])
system_parts.insert(0,
"VOICE MODE: Respond naturally as if speaking aloud. "
"No markdown, bullet points, headers, or code blocks. Complete sentences only. "
f"{style_hint}\n\n"
)
context_meta: dict = {
"context_note_id": None,
"context_note_title": None,
+2 -1
View File
@@ -17,6 +17,7 @@ async def create_milestone(
title: str,
description: str | None = None,
order_index: int = 0,
status: str = "active",
) -> Milestone:
async with async_session() as session:
milestone = Milestone(
@@ -24,7 +25,7 @@ async def create_milestone(
project_id=project_id,
title=title,
description=description,
status="active",
status=status,
order_index=order_index,
)
session.add(milestone)
+2 -1
View File
@@ -17,6 +17,7 @@ async def create_project(
description: str = "",
goal: str = "",
color: str | None = None,
status: str = "active",
) -> Project:
async with async_session() as session:
project = Project(
@@ -25,7 +26,7 @@ async def create_project(
description=description,
goal=goal,
color=color,
status="active",
status=status,
)
session.add(project)
await session.commit()
+4
View File
@@ -55,6 +55,10 @@ async def fetch_and_cache_feed(feed_id: int, url: str) -> int:
Fetch a feed URL, parse it, and upsert new items into rss_items.
Returns the number of new items stored.
"""
scheme = url.split("://")[0].lower() if "://" in url else ""
if scheme not in ("http", "https"):
logger.warning("Blocked RSS fetch with non-http(s) scheme: %s", url[:80])
return 0
try:
parsed = await _parse_feed(url)
except Exception:
+82
View File
@@ -0,0 +1,82 @@
"""Speech-to-text service using faster-whisper (in-process, CPU/GPU)."""
import asyncio
import logging
import tempfile
import time
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from faster_whisper import WhisperModel
logger = logging.getLogger(__name__)
_model: "WhisperModel | None" = None
_model_lock = asyncio.Lock()
_load_error: str | None = None
async def load_stt_model() -> None:
"""Load the Whisper model. Called once at startup when voice is enabled."""
global _model, _load_error
from fabledassistant.services.voice_config import get_stt_model, is_voice_enabled
if not await is_voice_enabled():
return
async with _model_lock:
if _model is not None:
return
try:
from faster_whisper import WhisperModel
model_name = await get_stt_model()
logger.info("Loading Whisper STT model '%s'...", model_name)
loop = asyncio.get_running_loop()
_model = await loop.run_in_executor(
None,
lambda: WhisperModel(model_name, device="cpu", compute_type="int8"),
)
logger.info("Whisper STT model '%s' loaded", model_name)
except Exception:
_load_error = "Failed to load Whisper STT model"
logger.exception(_load_error)
async def reload_stt_model() -> None:
"""Unload the current model and reload with the current config. Safe to call at runtime."""
global _model, _load_error
async with _model_lock:
_model = None
_load_error = None
await load_stt_model()
def stt_available() -> bool:
return _model is not None
async def transcribe(audio_bytes: bytes, mime_type: str = "audio/webm") -> str:
"""Transcribe audio bytes to text. Runs the model in a thread executor."""
if _model is None:
raise RuntimeError("STT model not loaded")
suffix = ".webm"
if "ogg" in mime_type:
suffix = ".ogg"
elif "wav" in mime_type:
suffix = ".wav"
elif "mp4" in mime_type or "m4a" in mime_type:
suffix = ".mp4"
def _run() -> str:
with tempfile.NamedTemporaryFile(suffix=suffix, delete=True) as f:
f.write(audio_bytes)
f.flush()
t0 = time.monotonic()
segments, _ = _model.transcribe(f.name, beam_size=5) # type: ignore[union-attr]
text = " ".join(seg.text.strip() for seg in segments).strip()
logger.debug("STT transcription took %.2fs", time.monotonic() - t0)
return text
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, _run)
+103
View File
@@ -0,0 +1,103 @@
"""Text-to-speech service using Kokoro TTS (in-process)."""
import asyncio
import io
import logging
import time
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from kokoro import KPipeline
logger = logging.getLogger(__name__)
_pipeline: "KPipeline | None" = None
_pipeline_lock = asyncio.Lock()
_load_error: str | None = None
# Static list of supported Kokoro voice IDs and display labels
_VOICES: list[dict] = [
{"id": "af_heart", "label": "Heart (American Female, warm)"},
{"id": "af_bella", "label": "Bella (American Female, expressive)"},
{"id": "af_nicole", "label": "Nicole (American Female, intimate)"},
{"id": "af_sarah", "label": "Sarah (American Female, clear)"},
{"id": "af_sky", "label": "Sky (American Female, bright)"},
{"id": "am_adam", "label": "Adam (American Male, neutral)"},
{"id": "am_michael", "label": "Michael (American Male, deep)"},
{"id": "bf_emma", "label": "Emma (British Female)"},
{"id": "bf_isabella", "label": "Isabella (British Female, formal)"},
{"id": "bm_george", "label": "George (British Male)"},
{"id": "bm_lewis", "label": "Lewis (British Male, casual)"},
]
async def load_tts_model() -> None:
"""Load the Kokoro pipeline. Called once at startup when voice is enabled."""
global _pipeline, _load_error
from fabledassistant.services.voice_config import is_voice_enabled
if not await is_voice_enabled():
return
async with _pipeline_lock:
if _pipeline is not None:
return
try:
from kokoro import KPipeline
logger.info("Loading Kokoro TTS pipeline...")
loop = asyncio.get_running_loop()
_pipeline = await loop.run_in_executor(
None,
lambda: KPipeline(lang_code="a"), # "a" = American English
)
logger.info("Kokoro TTS pipeline loaded")
except Exception:
_load_error = "Failed to load Kokoro TTS pipeline"
logger.exception(_load_error)
async def reload_tts_model() -> None:
"""Unload and reload the TTS pipeline. Safe to call at runtime."""
global _pipeline, _load_error
async with _pipeline_lock:
_pipeline = None
_load_error = None
await load_tts_model()
def tts_available() -> bool:
return _pipeline is not None
def list_voices() -> list[dict]:
return _VOICES
async def synthesise(text: str, voice: str = "af_heart", speed: float = 1.0) -> bytes:
"""Synthesise text to WAV bytes (24kHz, 16-bit mono). Runs in executor."""
if _pipeline is None:
raise RuntimeError("TTS pipeline not loaded")
speed = max(0.7, min(1.3, speed))
def _run() -> bytes:
import numpy as np
import soundfile as sf
t0 = time.monotonic()
audio_chunks: list = []
for _, _, audio in _pipeline(text, voice=voice, speed=speed): # type: ignore[misc]
if audio is not None:
audio_chunks.append(audio)
if not audio_chunks:
return b""
combined = np.concatenate(audio_chunks)
buf = io.BytesIO()
sf.write(buf, combined, samplerate=24000, format="WAV", subtype="PCM_16")
logger.debug("TTS synthesis took %.2fs for %d chars", time.monotonic() - t0, len(text))
return buf.getvalue()
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, _run)
@@ -0,0 +1,40 @@
"""System-wide voice configuration stored in the admin user's settings row.
Falls back to Config env vars so operators can still seed defaults via the
environment without touching the UI.
"""
from sqlalchemy import select
from fabledassistant.config import Config
from fabledassistant.models import async_session
from fabledassistant.models.setting import Setting
from fabledassistant.models.user import User
VOICE_SETTING_KEYS = ("voice_enabled", "voice_stt_model")
async def get_voice_config() -> dict[str, str]:
"""Return voice config, preferring DB values over env-var defaults."""
config: dict[str, str] = {
"voice_enabled": "true" if Config.VOICE_ENABLED else "false",
"voice_stt_model": Config.STT_MODEL,
}
async with async_session() as session:
result = await session.execute(
select(Setting)
.join(User, Setting.user_id == User.id)
.where(User.role == "admin", Setting.key.in_(VOICE_SETTING_KEYS))
)
for setting in result.scalars().all():
config[setting.key] = setting.value
return config
async def is_voice_enabled() -> bool:
config = await get_voice_config()
return config.get("voice_enabled", "false").lower() in ("1", "true", "yes")
async def get_stt_model() -> str:
config = await get_voice_config()
return config.get("voice_stt_model", Config.STT_MODEL)