From 3581cc15824d5b9d97d88246d222c7d83e66a0d9 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sun, 29 Mar 2026 18:17:58 -0400 Subject: [PATCH] docs: add voice S2S design spec Covers STT (faster-whisper), TTS (Kokoro), per-user settings, all new/modified files, audio format decisions, and 4-phase implementation plan. Co-Authored-By: Claude Sonnet 4.6 --- docs/2026-03-29-voice-s2s-design.md | 200 ++++++++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 docs/2026-03-29-voice-s2s-design.md diff --git a/docs/2026-03-29-voice-s2s-design.md b/docs/2026-03-29-voice-s2s-design.md new file mode 100644 index 0000000..3e0e88d --- /dev/null +++ b/docs/2026-03-29-voice-s2s-design.md @@ -0,0 +1,200 @@ +# Speech-to-Speech (S2S) Design Spec + +**Branch:** `feature/voice-s2s` +**Date:** 2026-03-29 +**Status:** Approved for implementation + +--- + +## Decisions + +- **STT:** faster-whisper (in-process Python, model size configurable via `STT_MODEL` env var) +- **TTS:** Kokoro TTS (in-process Python, voice/speed/style configurable per-user) +- **Input mode:** Push-to-talk (Phase 1); VAD deferred +- **No browser STT/TTS fallbacks** — self-hosted only, data stays on-server +- **No Android app** — web only for this implementation + +--- + +## New Env Vars (`config.py`) + +| Var | Default | Description | +|---|---|---| +| `VOICE_ENABLED` | `false` | Feature flag — opt-in | +| `STT_BACKEND` | `faster-whisper` | Only supported value currently | +| `STT_MODEL` | `base.en` | `tiny.en` / `base.en` / `small.en` / `medium.en` | +| `TTS_BACKEND` | `kokoro` | Only supported value currently | + +--- + +## Per-User Settings (stored in `settings` table) + +| Key | Default | Description | +|---|---|---| +| `voice_tts_voice` | `af_heart` | Kokoro voice ID | +| `voice_tts_speed` | `1.0` | Speech rate, 0.7–1.3 | +| `voice_speech_style` | `conversational` | `conversational` / `concise` / `detailed` | + +--- + +## New Backend Files + +### `src/fabledassistant/services/stt.py` +Lazy singleton `WhisperModel` loader. Public API: +- `load_stt_model()` — called at startup via `asyncio.create_task` +- `transcribe(audio_bytes, mime_type) -> str` — runs in `run_in_executor`; writes bytes to `NamedTemporaryFile`, returns concatenated segment text +- `stt_available() -> bool` + +### `src/fabledassistant/services/tts.py` +Lazy singleton `KPipeline` loader. Public API: +- `load_tts_model()` — called at startup +- `synthesise(text, voice, speed) -> bytes` — runs in `run_in_executor`; returns WAV bytes (24kHz, 16-bit mono) +- `list_voices() -> list[dict]` — returns static list of known Kokoro voice IDs + labels +- `tts_available() -> bool` + +### `src/fabledassistant/routes/voice.py` +Blueprint at `/api/voice`, all routes `@login_required`. + +| Endpoint | Method | Description | +|---|---|---| +| `/api/voice/status` | GET | STT/TTS availability; `enabled` false if `VOICE_ENABLED=false` | +| `/api/voice/voices` | GET | List available Kokoro voices | +| `/api/voice/transcribe` | POST | multipart `audio` field → `{"transcript": "...", "duration_ms": 123}` | +| `/api/voice/synthesise` | POST | `{"text", "voice", "speed"}` → WAV bytes | + +--- + +## Modified Backend Files + +### `src/fabledassistant/app.py` +- Register `voice_bp` blueprint +- In `startup()`: `asyncio.create_task(load_stt_model())` + `asyncio.create_task(load_tts_model())` when `VOICE_ENABLED` + +### `src/fabledassistant/config.py` +- Add 4 new env var attributes +- Add validation in `validate()` + +### `src/fabledassistant/services/llm.py` +- Add `voice_mode: bool = False` and `voice_speech_style: str = "conversational"` to `build_context()` +- When `voice_mode=True`, prepend: *"Respond naturally as if speaking aloud. No markdown, bullet points, headers, or code blocks. Complete sentences only."* +- Append style modifier based on `voice_speech_style` + +### `src/fabledassistant/services/generation_task.py` +- Add `voice_mode: bool = False` to `run_generation()` +- Read `voice_speech_style` from settings when voice_mode; pass both to `build_context()` + +### `src/fabledassistant/routes/chat.py` +- Allow `"voice"` in `conversation_type` whitelist + +### `src/fabledassistant/services/chat.py` +- Exclude `conversation_type == "voice"` from auto-cleanup retention + +--- + +## New Frontend Files + +### `frontend/src/composables/useVoiceRecorder.ts` +Wraps `MediaRecorder`. Exports: `recording`, `error`, `isSupported`, `startRecording()`, `stopRecording() -> Promise`. + +### `frontend/src/composables/useVoiceAudio.ts` +Wraps `AudioContext`. Exports: `playing`, `isSupported`, `play(blob)`, `stop()`. + +### `frontend/src/components/VoiceOverlay.vue` +Floating PTT button (fixed bottom-right). Creates/reuses a `"voice"` conversation. Full flow: record → transcribe → send → stream → synthesise → play. Space bar hotkey (from `App.vue`). Mounted globally in `App.vue`. + +--- + +## Modified Frontend Files + +### `frontend/src/api/client.ts` +Add: `transcribeAudio(blob)`, `synthesiseSpeech(text, voice?, speed?)`, `getVoiceStatus()`, `getVoiceList()` + +### `frontend/src/views/BriefingView.vue` +- "Listen" button: reads latest assistant message aloud via TTS +- Mic button in input bar: PTT → transcribe → auto-fill input → send +- Auto-TTS on assistant response when in listen mode + +### `frontend/src/views/SettingsView.vue` +- New "Voice" tab: voice dropdown, speed slider, speech style radio +- Loads from `/api/settings`, saves via `PUT /api/settings` + +### `frontend/src/App.vue` +- Mount `` +- Space bar → `"voice:ptt-toggle"` custom event + +--- + +## Audio Format + +| Direction | Format | Rationale | +|---|---|---| +| Browser → Server | WebM/Opus | Native `MediaRecorder` output; no re-encoding | +| Server → Browser | WAV (24kHz, 16-bit mono) | Kokoro native; no re-encoding; `decodeAudioData` compatible | + +--- + +## Dependencies to Add (`pyproject.toml`) + +```toml +[project.optional-dependencies] +voice = [ + "faster-whisper>=1.0", + "kokoro>=0.9", + "soundfile>=0.12", +] +``` + +Install unconditionally in Docker (activated by `VOICE_ENABLED` at runtime): +```dockerfile +RUN pip install faster-whisper kokoro soundfile +``` + +--- + +## Database Migration + +No schema changes required. `conversation_type` is unconstrained TEXT. Voice settings use existing key-value `settings` table. Optional no-op migration `0034_voice_conversation_type.py` for audit trail. + +--- + +## Implementation Phases + +### Phase 1 — Backend services + routes +1. Add env vars to `config.py` +2. Create `services/stt.py` (faster-whisper) +3. Create `services/tts.py` (Kokoro) +4. Create `routes/voice.py` (4 endpoints) +5. Wire model loading into `app.py` startup +6. Add `voice_mode` to `build_context()` + `run_generation()` +7. Allow `"voice"` conversation type in chat route + cleanup exclusion + +### Phase 2 — BriefingView listen + voice follow-up +1. Create `useVoiceRecorder.ts` +2. Create `useVoiceAudio.ts` +3. Add voice API functions to `client.ts` +4. Add "Listen" button + mic button to `BriefingView.vue` + +### Phase 3 — VoiceOverlay for general voice chat +1. Create `VoiceOverlay.vue` +2. Mount in `App.vue` + Space bar hotkey + +### Phase 4 — Settings UI +1. Add "Voice" tab to `SettingsView.vue` + +--- + +## Kokoro Voice Reference + +| ID | Character | +|---|---| +| `af_heart` | American female, warm (recommended default) | +| `af_bella` | American female, expressive | +| `af_nicole` | American female, breathy/intimate | +| `af_sarah` | American female, clear | +| `af_sky` | American female, bright | +| `am_adam` | American male, neutral | +| `am_michael` | American male, deeper | +| `bf_emma` | British female | +| `bf_isabella` | British female, formal | +| `bm_george` | British male | +| `bm_lewis` | British male, casual |