Files
FabledScribe/docs/superpowers/plans/2026-04-03-streaming-tts.md
T
2026-04-03 12:43:28 -04:00

595 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Streaming TTS Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Start playing TTS audio during LLM generation by splitting responses into sentences and synthesizing each sentence as it completes, rather than waiting for the full response.
**Architecture:** A new `useStreamingTts` composable watches `streamingContent` for sentence boundaries, fires per-sentence `synthesiseSpeech` requests concurrently, and plays audio in strict insertion order using `useVoiceAudio`. ChatView, BriefingView, and WorkspaceView all use this composable, replacing their current post-stream speak logic.
**Tech Stack:** Vue 3 Composition API, TypeScript, `useVoiceAudio` (existing), `synthesiseSpeech` from `api/client.ts` (existing), no backend changes.
---
## File Map
| Action | File | Responsibility |
|--------|------|----------------|
| **Create** | `frontend/src/composables/useStreamingTts.ts` | All streaming TTS logic: sentence splitting, TTS queuing, ordered playback |
| **Modify** | `frontend/src/views/ChatView.vue` | Replace `speakLastAssistantMessage` + old watch with `useStreamingTts` |
| **Modify** | `frontend/src/views/BriefingView.vue` | Replace `speakText` + `listenToLatest` + old watch with `useStreamingTts` |
| **Modify** | `frontend/src/views/WorkspaceView.vue` | Add listen mode toggle button + `useStreamingTts` |
---
## Task 1: Create `useStreamingTts` composable
**Files:**
- Create: `frontend/src/composables/useStreamingTts.ts`
- [ ] **Step 1: Create the composable**
Create `frontend/src/composables/useStreamingTts.ts` with the full implementation:
```typescript
import { ref, watch, computed } from 'vue'
import type { Ref, ComputedRef } from 'vue'
import { synthesiseSpeech } from '@/api/client'
import { useVoiceAudio } from '@/composables/useVoiceAudio'
/** Minimum stripped character count to bother synthesizing. */
const MIN_CHARS = 3
/** Matches sentence-terminal punctuation followed by whitespace or end-of-string. */
const SENTENCE_BOUNDARY = /[.!?]+(?=\s|$)/
function stripMarkdown(text: string): string {
return text
.replace(/```[\s\S]*?```/g, '')
.replace(/`[^`]+`/g, (m) => m.slice(1, -1))
.replace(/#{1,6}\s+/g, '')
.replace(/\*\*([^*]+)\*\*/g, '$1')
.replace(/\*([^*]+)\*/g, '$1')
.replace(/\[([^\]]+)\]\([^)]+\)/g, '$1')
.replace(/^\s*[-*+]\s+/gm, '')
.replace(/\n{2,}/g, ' ')
.trim()
}
/**
* Extract completed sentences from `text` using SENTENCE_BOUNDARY.
* Returns the sentences found and the unconsumed remainder.
*/
function extractSentences(text: string): { sentences: string[]; remainder: string } {
const sentences: string[] = []
let remaining = text
let match: RegExpExecArray | null
while ((match = SENTENCE_BOUNDARY.exec(remaining)) !== null) {
const boundary = match.index + match[0].length
const sentence = remaining.slice(0, boundary).trim()
if (sentence) sentences.push(sentence)
remaining = remaining.slice(boundary)
}
return { sentences, remainder: remaining }
}
export interface UseStreamingTtsOptions {
streamingContent: Ref<string> | ComputedRef<string>
streaming: Ref<boolean> | ComputedRef<boolean>
enabled: Ref<boolean> | ComputedRef<boolean>
}
export interface UseStreamingTtsReturn {
/** True while any synthesis request is in-flight or audio is playing. */
speaking: ComputedRef<boolean>
/** Cancel all in-flight synthesis/playback and clear the queue. */
stop: () => void
}
export function useStreamingTts(options: UseStreamingTtsOptions): UseStreamingTtsReturn {
const { streamingContent, streaming, enabled } = options
const audio = useVoiceAudio()
let sentenceBuffer = ''
let lastSeenLength = 0
let abortId = 0
let playQueue: Promise<void> = Promise.resolve()
const pendingCount = ref(0)
const speaking = computed(() => pendingCount.value > 0 || audio.playing.value)
function stop(): void {
abortId++
sentenceBuffer = ''
lastSeenLength = 0
playQueue = Promise.resolve()
audio.stop()
pendingCount.value = 0
}
async function enqueueSentence(sentence: string, myAbortId: number): Promise<void> {
const stripped = stripMarkdown(sentence)
if (stripped.length < MIN_CHARS) return
pendingCount.value++
let blob: Blob | null = null
try {
blob = await synthesiseSpeech(stripped)
} catch (e) {
console.warn('[StreamingTTS] Synthesis failed, retrying sentence', { sentence: stripped, error: e })
try {
blob = await synthesiseSpeech(stripped)
} catch (e2) {
console.warn('[StreamingTTS] Retry also failed, skipping sentence', { sentence: stripped, error: e2 })
}
} finally {
pendingCount.value--
}
if (!blob) return
// Capture blob for the closure — TS can't narrow after async gap
const resolvedBlob = blob
playQueue = playQueue.then(async () => {
if (abortId !== myAbortId) return
await audio.play(resolvedBlob)
})
}
function dispatchBuffer(flush: boolean): void {
if (!enabled.value) return
const myAbortId = abortId
const { sentences, remainder } = extractSentences(sentenceBuffer)
sentenceBuffer = flush ? '' : remainder
for (const sentence of sentences) {
enqueueSentence(sentence, myAbortId)
}
if (flush && remainder.trim().length >= MIN_CHARS) {
enqueueSentence(remainder.trim(), myAbortId)
}
}
// Watch accumulating content — extract new characters since last check
watch(streamingContent, (newContent) => {
if (!enabled.value) return
const delta = newContent.slice(lastSeenLength)
lastSeenLength = newContent.length
sentenceBuffer += delta
dispatchBuffer(false)
})
// Watch streaming flag — stop on new message start, flush on end
watch(streaming, (isStreaming) => {
if (!enabled.value) return
if (isStreaming) {
// New message starting — cancel previous response's audio
stop()
} else {
// Stream ended — flush any remaining fragment
dispatchBuffer(true)
lastSeenLength = 0
}
})
return { speaking, stop }
}
```
- [ ] **Step 2: TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```
Expected: no errors mentioning `useStreamingTts.ts`.
- [ ] **Step 3: Commit**
```bash
git add frontend/src/composables/useStreamingTts.ts
git commit -m "feat(tts): add useStreamingTts composable for sentence-level streaming"
```
---
## Task 2: Update ChatView
**Files:**
- Modify: `frontend/src/views/ChatView.vue`
Current TTS code to remove (lines ~3566):
```typescript
// REMOVE these:
const synthesising = ref(false);
async function speakLastAssistantMessage() { ... } // entire function
watch(() => store.streaming, async (streaming) => {
if (!streaming && listenMode.value && voiceTtsEnabled.value) {
await new Promise((r) => setTimeout(r, 200));
await speakLastAssistantMessage();
}
});
```
Also remove the `synthesiseSpeech` import from `@/api/client` (it is no longer called directly in this file).
- [ ] **Step 1: Add import and replace TTS logic**
In `frontend/src/views/ChatView.vue`:
1. Add to imports at the top of `<script setup>`:
```typescript
import { useStreamingTts } from "@/composables/useStreamingTts";
```
2. Remove `synthesiseSpeech` from the `@/api/client` import line (keep other imports like `apiGet`, `transcribeAudio`).
3. Remove `const synthesising = ref(false);` (line ~35).
4. Remove the entire `speakLastAssistantMessage` function (lines ~3859).
5. Remove the `watch(() => store.streaming, ...)` block that called `speakLastAssistantMessage` (lines ~6166).
6. Add after `const listenMode = useListenMode();`:
```typescript
const tts = useStreamingTts({
streamingContent: computed(() => store.streamingContent),
streaming: computed(() => store.streaming),
enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
});
```
- [ ] **Step 2: Update template references**
In the ChatView template, replace every occurrence of `synthesising` with `tts.speaking.value`:
Find (line ~919):
```html
:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': synthesising || audio.playing.value }"
```
Replace with:
```html
:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': tts.speaking.value }"
```
Find (line ~920):
```html
@click="listenMode = !listenMode; if (listenMode) speakLastAssistantMessage()"
```
Replace with:
```html
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"
```
Find (line ~924):
```html
<svg v-if="!synthesising && !audio.playing.value" ...>
```
Replace with:
```html
<svg v-if="!tts.speaking.value" ...>
```
Note: the `audio` variable (`useVoiceAudio()`) is still used for the volume slider and PTT stop — do NOT remove it.
- [ ] **Step 3: TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```
Expected: no errors.
- [ ] **Step 4: Commit**
```bash
git add frontend/src/views/ChatView.vue
git commit -m "feat(tts): wire useStreamingTts into ChatView"
```
---
## Task 3: Update BriefingView
**Files:**
- Modify: `frontend/src/views/BriefingView.vue`
Current TTS code to remove:
```typescript
// REMOVE:
const synthesising = ref(false)
async function speakText(text: string) { ... } // entire function
async function listenToLatest() { ... } // entire function
// REMOVE this watch block (the TTS one — keep the other streaming watch):
watch(() => chatStore.streaming, async (streaming) => {
if (!streaming && listenMode.value && voiceTtsEnabled.value) {
await new Promise((r) => setTimeout(r, 200))
await listenToLatest()
}
})
```
Note: BriefingView has **two** `watch(() => chatStore.streaming, ...)` blocks. Keep the first one (lines ~152156, which refreshes messages). Remove only the TTS one (lines ~327332).
Also remove the `synthesiseSpeech` import from `@/api/client`.
- [ ] **Step 1: Add import and replace TTS logic**
In `frontend/src/views/BriefingView.vue`:
1. Add to imports:
```typescript
import { useStreamingTts } from '@/composables/useStreamingTts'
```
2. Remove `synthesiseSpeech` from the `@/api/client` import line.
3. Remove `const synthesising = ref(false)`.
4. Remove the entire `speakText` function.
5. Remove the entire `listenToLatest` function.
6. Remove the TTS `watch(() => chatStore.streaming, ...)` block (the one that calls `listenToLatest`).
7. Add after `const listenMode = useListenMode()`:
```typescript
const tts = useStreamingTts({
streamingContent: computed(() => chatStore.streamingContent),
streaming: computed(() => chatStore.streaming),
enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})
```
- [ ] **Step 2: Update template references**
Find the listen toggle button in the template. Replace `synthesising` references:
```html
<!-- Before -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': synthesising || audio.playing.value }"
@click="listenMode ? (listenMode = false) : (listenMode = true, listenToLatest())"
<!-- After -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': tts.speaking.value }"
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"
```
Find the stop button:
```html
<!-- Before -->
v-if="voiceTtsEnabled && (synthesising || audio.playing.value)"
@click="audio.stop(); synthesising = false"
<!-- After -->
v-if="voiceTtsEnabled && tts.speaking.value"
@click="tts.stop()"
```
Find the spinner SVG condition:
```html
<!-- Before -->
<svg v-if="!synthesising && !audio.playing.value" ...>
<!-- After -->
<svg v-if="!tts.speaking.value" ...>
```
- [ ] **Step 3: TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```
Expected: no errors.
- [ ] **Step 4: Commit**
```bash
git add frontend/src/views/BriefingView.vue
git commit -m "feat(tts): wire useStreamingTts into BriefingView"
```
---
## Task 4: Add streaming TTS to WorkspaceView
**Files:**
- Modify: `frontend/src/views/WorkspaceView.vue`
WorkspaceView has no TTS today. We add: listen mode toggle, `useStreamingTts`, and the listen button in the chat input toolbar.
- [ ] **Step 1: Add imports and composable**
In `frontend/src/views/WorkspaceView.vue`, add to the import block at the top of `<script setup>`:
```typescript
import { useListenMode } from '@/composables/useListenMode'
import { useStreamingTts } from '@/composables/useStreamingTts'
import { useVoiceAudio } from '@/composables/useVoiceAudio'
```
After the existing store setup code (after `const settingsStore = useSettingsStore()`), add:
```typescript
const listenMode = useListenMode()
const voiceTtsEnabled = computed(() => settingsStore.voiceTtsReady)
const audio = useVoiceAudio()
const tts = useStreamingTts({
streamingContent: computed(() => chatStore.streamingContent),
streaming: computed(() => chatStore.streaming),
enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})
```
- [ ] **Step 2: Add listen mode button to template**
In the `<div class="chat-input-area">` section (around line 365), add the listen button before the abort/send button:
```html
<div class="chat-input-area">
<textarea
ref="inputEl"
v-model="messageInput"
class="chat-input"
:placeholder="chatStore.streaming ? 'Type to queue next message… (Enter to queue)' : 'Message the agent… (Enter to send)'"
rows="1"
@keydown="onInputKeydown"
@input="autoResize"
></textarea>
<!-- Listen mode toggle (TTS) -->
<button
v-if="voiceTtsEnabled"
class="btn-listen-ws"
:class="{ 'btn-listen-ws--active': listenMode, 'btn-listen-ws--busy': tts.speaking.value }"
:title="listenMode ? 'Stop auto-read' : 'Read responses aloud'"
aria-label="Toggle listen mode"
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"
>
<svg v-if="!tts.speaking.value" width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
<path d="M3 9v6h4l5 5V4L7 9H3zm13.5 3c0-1.77-1.02-3.29-2.5-4.03v8.05c1.48-.73 2.5-2.25 2.5-4.02zM14 3.23v2.06c2.89.86 5 3.54 5 6.71s-2.11 5.85-5 6.71v2.06c4.01-.91 7-4.49 7-8.77s-2.99-7.86-7-8.77z"/>
</svg>
<svg v-else width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
<path d="M18 12c0-1.77-1.02-3.29-2.5-4.03v2.21l2.45 2.45c.03-.2.05-.41.05-.63zm2.5 0c0 .94-.2 1.82-.54 2.64l1.51 1.51C21.8 14.82 22 13.43 22 12c0-4.28-2.99-7.86-7-8.77v2.06c2.89.86 5 3.54 5 6.71zM4.27 3L3 4.27 7.73 9H3v6h4l5 5v-6.73l4.25 4.25c-.67.52-1.42.93-2.25 1.18v2.06c1.38-.31 2.63-.95 3.69-1.81L19.73 21 21 19.73l-9-9L4.27 3zM12 4L9.91 6.09 12 8.18V4z"/>
</svg>
</button>
<button
v-if="chatStore.streaming"
class="btn-abort"
title="Stop generation"
@click="chatStore.cancelGeneration()"
>
■ Stop
</button>
<button
v-else
class="btn-send"
:disabled="!messageInput.trim()"
@click="sendMessage"
>
Send
</button>
</div>
```
- [ ] **Step 3: Add CSS for the listen button**
In the `<style>` block, add:
```css
.btn-listen-ws {
flex-shrink: 0;
display: flex;
align-items: center;
justify-content: center;
width: 2rem;
height: 2rem;
border: 1px solid var(--color-border);
border-radius: var(--radius-md);
background: transparent;
color: var(--color-text-muted);
cursor: pointer;
transition: color 0.15s, background 0.15s, border-color 0.15s;
}
.btn-listen-ws:hover {
color: var(--color-text);
border-color: var(--color-primary);
}
.btn-listen-ws--active {
color: var(--color-primary);
border-color: var(--color-primary);
background: color-mix(in srgb, var(--color-primary) 10%, transparent);
}
.btn-listen-ws--busy {
color: var(--color-primary);
animation: pulse 1.2s ease-in-out infinite;
}
```
- [ ] **Step 4: TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```
Expected: no errors.
- [ ] **Step 5: Commit**
```bash
git add frontend/src/views/WorkspaceView.vue
git commit -m "feat(tts): add streaming TTS listen mode to WorkspaceView"
```
---
## Task 5: Final integration check and push
- [ ] **Step 1: Full TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1
```
Expected: zero errors.
- [ ] **Step 2: Verify no dead imports remain**
```bash
grep -n "synthesiseSpeech\|speakLastAssistantMessage\|speakText\|listenToLatest\|synthesising" \
frontend/src/views/ChatView.vue \
frontend/src/views/BriefingView.vue \
frontend/src/views/WorkspaceView.vue
```
Expected: no matches (all replaced).
- [ ] **Step 3: Manual smoke test**
1. Enable voice in Admin → Config
2. Open Chat, enable listen mode (speaker icon)
3. Send a message and watch: audio should begin playing the first sentence while the LLM is still streaming the response
4. Send another message mid-playback — previous audio should stop immediately
5. Toggle listen mode off mid-response — audio stops, `tts.stop()` called
6. Repeat in `/briefing` and `/workspace/:id`
- [ ] **Step 4: Push**
```bash
git push origin dev
```
---
## Self-Review
**Spec coverage check:**
- ✅ Starts playing during generation (sentence-level queue, fires on each boundary)
- ✅ Automatic when listen mode on (enabled computed = listenMode && voiceTtsEnabled)
- ✅ ChatView updated
- ✅ BriefingView updated
- ✅ WorkspaceView added
- ✅ One retry before skipping on failure
- ✅ Failures logged via `console.warn` with sentence text and error
-`stop()` on new message start (watch streaming → true)
- ✅ Flush remaining buffer on stream end (watch streaming → false)
- ✅ Fragments < 3 chars skipped
-`abortId` prevents stale playback after stop
**Type consistency:**
- `tts.speaking` is `ComputedRef<boolean>` — accessed as `tts.speaking.value` in templates ✅
- `tts.stop()` called consistently across all three views ✅
- `useStreamingTts` options match usage in all three call sites ✅
- `audio` variable kept in ChatView (used by volume slider) — not removed ✅