FabledScribe/docs/superpowers/specs/2026-04-03-streaming-tts-design.md

Streaming TTS Design

Date: 2026-04-03
Status: Approved

Goal

Start playing TTS audio during LLM generation rather than waiting for the full response to finish. When listen mode is on, the first sentence plays as soon as Kokoro finishes synthesizing it — while the LLM is still streaming the rest of the response.

Approach

Client-side sentence queuing composable. The frontend accumulates streaming tokens, detects sentence boundaries, fires per-sentence synthesis requests concurrently, and plays audio in strict insertion order. The existing /api/voice/synthesise backend endpoint is unchanged.

Architecture

useStreamingTts composable

File: frontend/src/composables/useStreamingTts.ts

Inputs:

• streamingContent: Ref<string> — the growing accumulated response text (e.g. store.streamingContent)
• streaming: Ref<boolean> — whether the LLM is currently generating
• enabled: Ref<boolean> — true when listen mode is on AND TTS is available

Exports:

• speaking: Ref<boolean> — true while any synthesis is in-flight or audio is playing
• stop() — cancels all pending synthesis/playback and clears the queue

Internal state:

• sentenceBuffer: string — accumulates characters since the last dispatched sentence
• lastSeenLength: number — tracks how far into streamingContent we've processed
• abortId: number — incremented on stop(); each queued promise checks the current id and bails if stale
• playQueue: Promise<void> — a chained promise that serializes audio playback in insertion order

Sentence detection:

• Regex: /[.!?]+(?=\s|$)/ — handles ..., ?!, multi-punctuation
• Triggered on every streamingContent change and on streaming flipping false (flush)
• Fragments < 3 characters after markdown stripping are skipped

Per-sentence pipeline:

1. Strip markdown (same logic as current speakLastAssistantMessage)
2. Fire synthesiseSpeech(sentence) immediately — runs concurrently with other sentences
3. On failure: one immediate retry. If retry also fails, skip silently and advance the queue
4. Resolved blob is inserted into the playback queue at its original position
5. Playback queue plays blobs strictly in insertion order via useVoiceAudio

Stream-end flush:

• When streaming flips false, any remaining sentenceBuffer content (fragment without terminal punctuation) is dispatched as a final sentence — covers responses that end without a period

Automatic reset:

• When streaming flips true (new message starting), stop() is called automatically to cancel any in-flight audio from the previous response before starting fresh

Views updated

View	Change
ChatView.vue	Replace speakLastAssistantMessage() + watch(streaming) + synthesising ref with useStreamingTts
BriefingView.vue	Replace speakText() + watch(streaming) + synthesising ref with useStreamingTts
WorkspaceView.vue	Add listen mode toggle button (same UI pattern as ChatView) + useStreamingTts wired to workspace chat stream

In all three views: the speaking export from useStreamingTts replaces the old synthesising || audio.playing.value checks for button busy state.

Backend

No changes. /api/voice/synthesise accepts shorter sentence-length strings without issue.

Error Handling

Scenario	Behavior
Synthesis fails for a sentence	One immediate retry; if retry fails, sentence is skipped, queue advances, and a console.warn is emitted with the sentence index and error
stop() called mid-queue	abortId incremented; all in-flight promises check id and discard their result
New message starts while audio playing	watch(streaming, true → ...) calls stop() before starting new queue
TTS unavailable or listen mode off	Composable is inert — watchers do nothing, no requests fired
Fragment < 3 chars after stripping	Skipped without a TTS request
Response ends without terminal punctuation	Remaining buffer flushed as final sentence on stream-end

Data Flow

LLM SSE chunks → store.streamingContent (grows)
                         ↓
              useStreamingTts watcher
                         ↓
              sentenceBuffer accumulation
                         ↓
         sentence boundary detected? → synthesiseSpeech(sentence) [concurrent]
                         ↓                        ↓ (fail → 1 retry → skip)
              playQueue.then(play blob)    resolved blob
                         ↓
              useVoiceAudio.play() [sequential]
                         ↓
                    audio output

Files Changed

• New: frontend/src/composables/useStreamingTts.ts
• Modified: frontend/src/views/ChatView.vue — swap TTS logic for composable
• Modified: frontend/src/views/BriefingView.vue — swap TTS logic for composable
• Modified: frontend/src/views/WorkspaceView.vue — add listen mode + composable