FabledScribe/docs/2026-03-29-voice-s2s-design.md

Speech-to-Speech (S2S) Design Spec

Branch: feature/voice-s2s Date: 2026-03-29 Status: Approved for implementation

---

Decisions

• STT: faster-whisper (in-process Python, model size configurable via STT_MODEL env var)
• TTS: Kokoro TTS (in-process Python, voice/speed/style configurable per-user)
• Input mode: Push-to-talk (Phase 1); VAD deferred
• No browser STT/TTS fallbacks — self-hosted only, data stays on-server
• No Android app — web only for this implementation

---

New Env Vars (config.py)

Var	Default	Description
VOICE_ENABLED	false	Feature flag — opt-in
STT_BACKEND	faster-whisper	Only supported value currently
STT_MODEL	base.en	tiny.en / base.en / small.en / medium.en
TTS_BACKEND	kokoro	Only supported value currently

---

Per-User Settings (stored in settings table)

Key	Default	Description
voice_tts_voice	af_heart	Kokoro voice ID
voice_tts_speed	1.0	Speech rate, 0.7–1.3
voice_speech_style	conversational	conversational / concise / detailed

---

New Backend Files

src/scribe/services/stt.py

Lazy singleton WhisperModel loader. Public API:

• load_stt_model() — called at startup via asyncio.create_task
• transcribe(audio_bytes, mime_type) -> str — runs in run_in_executor; writes bytes to NamedTemporaryFile, returns concatenated segment text
• stt_available() -> bool

src/scribe/services/tts.py

Lazy singleton KPipeline loader. Public API:

• load_tts_model() — called at startup
• synthesise(text, voice, speed) -> bytes — runs in run_in_executor; returns WAV bytes (24kHz, 16-bit mono)
• list_voices() -> list[dict] — returns static list of known Kokoro voice IDs + labels
• tts_available() -> bool

src/scribe/routes/voice.py

Blueprint at /api/voice, all routes @login_required.

Endpoint	Method	Description
/api/voice/status	GET	STT/TTS availability; enabled false if VOICE_ENABLED=false
/api/voice/voices	GET	List available Kokoro voices
/api/voice/transcribe	POST	multipart audio field → {"transcript": "...", "duration_ms": 123}
/api/voice/synthesise	POST	{"text", "voice", "speed"} → WAV bytes

---

Modified Backend Files

src/scribe/app.py

• Register voice_bp blueprint
• In startup(): asyncio.create_task(load_stt_model()) + asyncio.create_task(load_tts_model()) when VOICE_ENABLED

src/scribe/config.py

• Add 4 new env var attributes
• Add validation in validate()

src/scribe/services/llm.py

• Add voice_mode: bool = False and voice_speech_style: str = "conversational" to build_context()
• When voice_mode=True, prepend: "Respond naturally as if speaking aloud. No markdown, bullet points, headers, or code blocks. Complete sentences only."
• Append style modifier based on voice_speech_style

src/scribe/services/generation_task.py

• Add voice_mode: bool = False to run_generation()
• Read voice_speech_style from settings when voice_mode; pass both to build_context()

src/scribe/routes/chat.py

• Allow "voice" in conversation_type whitelist

src/scribe/services/chat.py

• Exclude conversation_type == "voice" from auto-cleanup retention

---

New Frontend Files

frontend/src/composables/useVoiceRecorder.ts

Wraps MediaRecorder. Exports: recording, error, isSupported, startRecording(), stopRecording() -> Promise<Blob>.

frontend/src/composables/useVoiceAudio.ts

Wraps AudioContext. Exports: playing, isSupported, play(blob), stop().

frontend/src/components/VoiceOverlay.vue

Floating PTT button (fixed bottom-right). Creates/reuses a "voice" conversation. Full flow: record → transcribe → send → stream → synthesise → play. Space bar hotkey (from App.vue). Mounted globally in App.vue.

---

Modified Frontend Files

frontend/src/api/client.ts

Add: transcribeAudio(blob), synthesiseSpeech(text, voice?, speed?), getVoiceStatus(), getVoiceList()

frontend/src/views/BriefingView.vue

• "Listen" button: reads latest assistant message aloud via TTS
• Mic button in input bar: PTT → transcribe → auto-fill input → send
• Auto-TTS on assistant response when in listen mode

frontend/src/views/SettingsView.vue

• New "Voice" tab: voice dropdown, speed slider, speech style radio
• Loads from /api/settings, saves via PUT /api/settings

frontend/src/App.vue

• Mount <VoiceOverlay />
• Space bar → "voice:ptt-toggle" custom event

---

Audio Format

Direction	Format	Rationale
Browser → Server	WebM/Opus	Native MediaRecorder output; no re-encoding
Server → Browser	WAV (24kHz, 16-bit mono)	Kokoro native; no re-encoding; decodeAudioData compatible

---

Dependencies to Add (pyproject.toml)

[project.optional-dependencies]
voice = [
    "faster-whisper>=1.0",
    "kokoro>=0.9",
    "soundfile>=0.12",
]

Install unconditionally in Docker (activated by VOICE_ENABLED at runtime):

RUN pip install faster-whisper kokoro soundfile

---

Database Migration

No schema changes required. conversation_type is unconstrained TEXT. Voice settings use existing key-value settings table. Optional no-op migration 0034_voice_conversation_type.py for audit trail.

---

Implementation Phases

Phase 1 — Backend services + routes

1. Add env vars to config.py
2. Create services/stt.py (faster-whisper)
3. Create services/tts.py (Kokoro)
4. Create routes/voice.py (4 endpoints)
5. Wire model loading into app.py startup
6. Add voice_mode to build_context() + run_generation()
7. Allow "voice" conversation type in chat route + cleanup exclusion

Phase 2 — BriefingView listen + voice follow-up

1. Create useVoiceRecorder.ts
2. Create useVoiceAudio.ts
3. Add voice API functions to client.ts
4. Add "Listen" button + mic button to BriefingView.vue

Phase 3 — VoiceOverlay for general voice chat

1. Create VoiceOverlay.vue
2. Mount in App.vue + Space bar hotkey

Phase 4 — Settings UI

1. Add "Voice" tab to SettingsView.vue

---

Kokoro Voice Reference

ID	Character
af_heart	American female, warm (recommended default)
af_bella	American female, expressive
af_nicole	American female, breathy/intimate
af_sarah	American female, clear
af_sky	American female, bright
am_adam	American male, neutral
am_michael	American male, deeper
bf_emma	British female
bf_isabella	British female, formal
bm_george	British male
bm_lewis	British male, casual