102 lines
4.9 KiB
Markdown
102 lines
4.9 KiB
Markdown
# Streaming TTS Design
|
|
|
|
**Date:** 2026-04-03
|
|
**Status:** Approved
|
|
|
|
## Goal
|
|
|
|
Start playing TTS audio during LLM generation rather than waiting for the full response to finish. When listen mode is on, the first sentence plays as soon as Kokoro finishes synthesizing it — while the LLM is still streaming the rest of the response.
|
|
|
|
## Approach
|
|
|
|
Client-side sentence queuing composable. The frontend accumulates streaming tokens, detects sentence boundaries, fires per-sentence synthesis requests concurrently, and plays audio in strict insertion order. The existing `/api/voice/synthesise` backend endpoint is unchanged.
|
|
|
|
## Architecture
|
|
|
|
### `useStreamingTts` composable
|
|
|
|
**File:** `frontend/src/composables/useStreamingTts.ts`
|
|
|
|
**Inputs:**
|
|
- `streamingContent: Ref<string>` — the growing accumulated response text (e.g. `store.streamingContent`)
|
|
- `streaming: Ref<boolean>` — whether the LLM is currently generating
|
|
- `enabled: Ref<boolean>` — `true` when listen mode is on AND TTS is available
|
|
|
|
**Exports:**
|
|
- `speaking: Ref<boolean>` — `true` while any synthesis is in-flight or audio is playing
|
|
- `stop()` — cancels all pending synthesis/playback and clears the queue
|
|
|
|
**Internal state:**
|
|
- `sentenceBuffer: string` — accumulates characters since the last dispatched sentence
|
|
- `lastSeenLength: number` — tracks how far into `streamingContent` we've processed
|
|
- `abortId: number` — incremented on `stop()`; each queued promise checks the current id and bails if stale
|
|
- `playQueue: Promise<void>` — a chained promise that serializes audio playback in insertion order
|
|
|
|
**Sentence detection:**
|
|
- Regex: `/[.!?]+(?=\s|$)/` — handles `...`, `?!`, multi-punctuation
|
|
- Triggered on every `streamingContent` change and on `streaming` flipping `false` (flush)
|
|
- Fragments < 3 characters after markdown stripping are skipped
|
|
|
|
**Per-sentence pipeline:**
|
|
1. Strip markdown (same logic as current `speakLastAssistantMessage`)
|
|
2. Fire `synthesiseSpeech(sentence)` immediately — runs concurrently with other sentences
|
|
3. On failure: one immediate retry. If retry also fails, skip silently and advance the queue
|
|
4. Resolved blob is inserted into the playback queue at its original position
|
|
5. Playback queue plays blobs strictly in insertion order via `useVoiceAudio`
|
|
|
|
**Stream-end flush:**
|
|
- When `streaming` flips `false`, any remaining `sentenceBuffer` content (fragment without terminal punctuation) is dispatched as a final sentence — covers responses that end without a period
|
|
|
|
**Automatic reset:**
|
|
- When `streaming` flips `true` (new message starting), `stop()` is called automatically to cancel any in-flight audio from the previous response before starting fresh
|
|
|
|
### Views updated
|
|
|
|
| View | Change |
|
|
|------|--------|
|
|
| `ChatView.vue` | Replace `speakLastAssistantMessage()` + `watch(streaming)` + `synthesising` ref with `useStreamingTts` |
|
|
| `BriefingView.vue` | Replace `speakText()` + `watch(streaming)` + `synthesising` ref with `useStreamingTts` |
|
|
| `WorkspaceView.vue` | Add listen mode toggle button (same UI pattern as ChatView) + `useStreamingTts` wired to workspace chat stream |
|
|
|
|
In all three views: the `speaking` export from `useStreamingTts` replaces the old `synthesising || audio.playing.value` checks for button busy state.
|
|
|
|
### Backend
|
|
|
|
No changes. `/api/voice/synthesise` accepts shorter sentence-length strings without issue.
|
|
|
|
## Error Handling
|
|
|
|
| Scenario | Behavior |
|
|
|----------|----------|
|
|
| Synthesis fails for a sentence | One immediate retry; if retry fails, sentence is skipped, queue advances, and a `console.warn` is emitted with the sentence index and error |
|
|
| `stop()` called mid-queue | `abortId` incremented; all in-flight promises check id and discard their result |
|
|
| New message starts while audio playing | `watch(streaming, true → ...)` calls `stop()` before starting new queue |
|
|
| TTS unavailable or listen mode off | Composable is inert — watchers do nothing, no requests fired |
|
|
| Fragment < 3 chars after stripping | Skipped without a TTS request |
|
|
| Response ends without terminal punctuation | Remaining buffer flushed as final sentence on stream-end |
|
|
|
|
## Data Flow
|
|
|
|
```
|
|
LLM SSE chunks → store.streamingContent (grows)
|
|
↓
|
|
useStreamingTts watcher
|
|
↓
|
|
sentenceBuffer accumulation
|
|
↓
|
|
sentence boundary detected? → synthesiseSpeech(sentence) [concurrent]
|
|
↓ ↓ (fail → 1 retry → skip)
|
|
playQueue.then(play blob) resolved blob
|
|
↓
|
|
useVoiceAudio.play() [sequential]
|
|
↓
|
|
audio output
|
|
```
|
|
|
|
## Files Changed
|
|
|
|
- **New:** `frontend/src/composables/useStreamingTts.ts`
|
|
- **Modified:** `frontend/src/views/ChatView.vue` — swap TTS logic for composable
|
|
- **Modified:** `frontend/src/views/BriefingView.vue` — swap TTS logic for composable
|
|
- **Modified:** `frontend/src/views/WorkspaceView.vue` — add listen mode + composable
|