bvandeusen
3581cc1582
Covers STT (faster-whisper), TTS (Kokoro), per-user settings, all new/modified files, audio format decisions, and 4-phase implementation plan. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
201 lines
6.8 KiB
Markdown
201 lines
6.8 KiB
Markdown
# Speech-to-Speech (S2S) Design Spec
|
||
|
||
**Branch:** `feature/voice-s2s`
|
||
**Date:** 2026-03-29
|
||
**Status:** Approved for implementation
|
||
|
||
---
|
||
|
||
## Decisions
|
||
|
||
- **STT:** faster-whisper (in-process Python, model size configurable via `STT_MODEL` env var)
|
||
- **TTS:** Kokoro TTS (in-process Python, voice/speed/style configurable per-user)
|
||
- **Input mode:** Push-to-talk (Phase 1); VAD deferred
|
||
- **No browser STT/TTS fallbacks** — self-hosted only, data stays on-server
|
||
- **No Android app** — web only for this implementation
|
||
|
||
---
|
||
|
||
## New Env Vars (`config.py`)
|
||
|
||
| Var | Default | Description |
|
||
|---|---|---|
|
||
| `VOICE_ENABLED` | `false` | Feature flag — opt-in |
|
||
| `STT_BACKEND` | `faster-whisper` | Only supported value currently |
|
||
| `STT_MODEL` | `base.en` | `tiny.en` / `base.en` / `small.en` / `medium.en` |
|
||
| `TTS_BACKEND` | `kokoro` | Only supported value currently |
|
||
|
||
---
|
||
|
||
## Per-User Settings (stored in `settings` table)
|
||
|
||
| Key | Default | Description |
|
||
|---|---|---|
|
||
| `voice_tts_voice` | `af_heart` | Kokoro voice ID |
|
||
| `voice_tts_speed` | `1.0` | Speech rate, 0.7–1.3 |
|
||
| `voice_speech_style` | `conversational` | `conversational` / `concise` / `detailed` |
|
||
|
||
---
|
||
|
||
## New Backend Files
|
||
|
||
### `src/fabledassistant/services/stt.py`
|
||
Lazy singleton `WhisperModel` loader. Public API:
|
||
- `load_stt_model()` — called at startup via `asyncio.create_task`
|
||
- `transcribe(audio_bytes, mime_type) -> str` — runs in `run_in_executor`; writes bytes to `NamedTemporaryFile`, returns concatenated segment text
|
||
- `stt_available() -> bool`
|
||
|
||
### `src/fabledassistant/services/tts.py`
|
||
Lazy singleton `KPipeline` loader. Public API:
|
||
- `load_tts_model()` — called at startup
|
||
- `synthesise(text, voice, speed) -> bytes` — runs in `run_in_executor`; returns WAV bytes (24kHz, 16-bit mono)
|
||
- `list_voices() -> list[dict]` — returns static list of known Kokoro voice IDs + labels
|
||
- `tts_available() -> bool`
|
||
|
||
### `src/fabledassistant/routes/voice.py`
|
||
Blueprint at `/api/voice`, all routes `@login_required`.
|
||
|
||
| Endpoint | Method | Description |
|
||
|---|---|---|
|
||
| `/api/voice/status` | GET | STT/TTS availability; `enabled` false if `VOICE_ENABLED=false` |
|
||
| `/api/voice/voices` | GET | List available Kokoro voices |
|
||
| `/api/voice/transcribe` | POST | multipart `audio` field → `{"transcript": "...", "duration_ms": 123}` |
|
||
| `/api/voice/synthesise` | POST | `{"text", "voice", "speed"}` → WAV bytes |
|
||
|
||
---
|
||
|
||
## Modified Backend Files
|
||
|
||
### `src/fabledassistant/app.py`
|
||
- Register `voice_bp` blueprint
|
||
- In `startup()`: `asyncio.create_task(load_stt_model())` + `asyncio.create_task(load_tts_model())` when `VOICE_ENABLED`
|
||
|
||
### `src/fabledassistant/config.py`
|
||
- Add 4 new env var attributes
|
||
- Add validation in `validate()`
|
||
|
||
### `src/fabledassistant/services/llm.py`
|
||
- Add `voice_mode: bool = False` and `voice_speech_style: str = "conversational"` to `build_context()`
|
||
- When `voice_mode=True`, prepend: *"Respond naturally as if speaking aloud. No markdown, bullet points, headers, or code blocks. Complete sentences only."*
|
||
- Append style modifier based on `voice_speech_style`
|
||
|
||
### `src/fabledassistant/services/generation_task.py`
|
||
- Add `voice_mode: bool = False` to `run_generation()`
|
||
- Read `voice_speech_style` from settings when voice_mode; pass both to `build_context()`
|
||
|
||
### `src/fabledassistant/routes/chat.py`
|
||
- Allow `"voice"` in `conversation_type` whitelist
|
||
|
||
### `src/fabledassistant/services/chat.py`
|
||
- Exclude `conversation_type == "voice"` from auto-cleanup retention
|
||
|
||
---
|
||
|
||
## New Frontend Files
|
||
|
||
### `frontend/src/composables/useVoiceRecorder.ts`
|
||
Wraps `MediaRecorder`. Exports: `recording`, `error`, `isSupported`, `startRecording()`, `stopRecording() -> Promise<Blob>`.
|
||
|
||
### `frontend/src/composables/useVoiceAudio.ts`
|
||
Wraps `AudioContext`. Exports: `playing`, `isSupported`, `play(blob)`, `stop()`.
|
||
|
||
### `frontend/src/components/VoiceOverlay.vue`
|
||
Floating PTT button (fixed bottom-right). Creates/reuses a `"voice"` conversation. Full flow: record → transcribe → send → stream → synthesise → play. Space bar hotkey (from `App.vue`). Mounted globally in `App.vue`.
|
||
|
||
---
|
||
|
||
## Modified Frontend Files
|
||
|
||
### `frontend/src/api/client.ts`
|
||
Add: `transcribeAudio(blob)`, `synthesiseSpeech(text, voice?, speed?)`, `getVoiceStatus()`, `getVoiceList()`
|
||
|
||
### `frontend/src/views/BriefingView.vue`
|
||
- "Listen" button: reads latest assistant message aloud via TTS
|
||
- Mic button in input bar: PTT → transcribe → auto-fill input → send
|
||
- Auto-TTS on assistant response when in listen mode
|
||
|
||
### `frontend/src/views/SettingsView.vue`
|
||
- New "Voice" tab: voice dropdown, speed slider, speech style radio
|
||
- Loads from `/api/settings`, saves via `PUT /api/settings`
|
||
|
||
### `frontend/src/App.vue`
|
||
- Mount `<VoiceOverlay />`
|
||
- Space bar → `"voice:ptt-toggle"` custom event
|
||
|
||
---
|
||
|
||
## Audio Format
|
||
|
||
| Direction | Format | Rationale |
|
||
|---|---|---|
|
||
| Browser → Server | WebM/Opus | Native `MediaRecorder` output; no re-encoding |
|
||
| Server → Browser | WAV (24kHz, 16-bit mono) | Kokoro native; no re-encoding; `decodeAudioData` compatible |
|
||
|
||
---
|
||
|
||
## Dependencies to Add (`pyproject.toml`)
|
||
|
||
```toml
|
||
[project.optional-dependencies]
|
||
voice = [
|
||
"faster-whisper>=1.0",
|
||
"kokoro>=0.9",
|
||
"soundfile>=0.12",
|
||
]
|
||
```
|
||
|
||
Install unconditionally in Docker (activated by `VOICE_ENABLED` at runtime):
|
||
```dockerfile
|
||
RUN pip install faster-whisper kokoro soundfile
|
||
```
|
||
|
||
---
|
||
|
||
## Database Migration
|
||
|
||
No schema changes required. `conversation_type` is unconstrained TEXT. Voice settings use existing key-value `settings` table. Optional no-op migration `0034_voice_conversation_type.py` for audit trail.
|
||
|
||
---
|
||
|
||
## Implementation Phases
|
||
|
||
### Phase 1 — Backend services + routes
|
||
1. Add env vars to `config.py`
|
||
2. Create `services/stt.py` (faster-whisper)
|
||
3. Create `services/tts.py` (Kokoro)
|
||
4. Create `routes/voice.py` (4 endpoints)
|
||
5. Wire model loading into `app.py` startup
|
||
6. Add `voice_mode` to `build_context()` + `run_generation()`
|
||
7. Allow `"voice"` conversation type in chat route + cleanup exclusion
|
||
|
||
### Phase 2 — BriefingView listen + voice follow-up
|
||
1. Create `useVoiceRecorder.ts`
|
||
2. Create `useVoiceAudio.ts`
|
||
3. Add voice API functions to `client.ts`
|
||
4. Add "Listen" button + mic button to `BriefingView.vue`
|
||
|
||
### Phase 3 — VoiceOverlay for general voice chat
|
||
1. Create `VoiceOverlay.vue`
|
||
2. Mount in `App.vue` + Space bar hotkey
|
||
|
||
### Phase 4 — Settings UI
|
||
1. Add "Voice" tab to `SettingsView.vue`
|
||
|
||
---
|
||
|
||
## Kokoro Voice Reference
|
||
|
||
| ID | Character |
|
||
|---|---|
|
||
| `af_heart` | American female, warm (recommended default) |
|
||
| `af_bella` | American female, expressive |
|
||
| `af_nicole` | American female, breathy/intimate |
|
||
| `af_sarah` | American female, clear |
|
||
| `af_sky` | American female, bright |
|
||
| `am_adam` | American male, neutral |
|
||
| `am_michael` | American male, deeper |
|
||
| `bf_emma` | British female |
|
||
| `bf_isabella` | British female, formal |
|
||
| `bm_george` | British male |
|
||
| `bm_lewis` | British male, casual |
|