FabledScribe/docs/superpowers/plans/2026-04-03-streaming-tts.md

Streaming TTS Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Start playing TTS audio during LLM generation by splitting responses into sentences and synthesizing each sentence as it completes, rather than waiting for the full response.

Architecture: A new useStreamingTts composable watches streamingContent for sentence boundaries, fires per-sentence synthesiseSpeech requests concurrently, and plays audio in strict insertion order using useVoiceAudio. ChatView, BriefingView, and WorkspaceView all use this composable, replacing their current post-stream speak logic.

Tech Stack: Vue 3 Composition API, TypeScript, useVoiceAudio (existing), synthesiseSpeech from api/client.ts (existing), no backend changes.

---

File Map

Action	File	Responsibility
Create	frontend/src/composables/useStreamingTts.ts	All streaming TTS logic: sentence splitting, TTS queuing, ordered playback
Modify	frontend/src/views/ChatView.vue	Replace speakLastAssistantMessage + old watch with useStreamingTts
Modify	frontend/src/views/BriefingView.vue	Replace speakText + listenToLatest + old watch with useStreamingTts
Modify	frontend/src/views/WorkspaceView.vue	Add listen mode toggle button + useStreamingTts

---

Task 1: Create useStreamingTts composable

Files:

• Create: frontend/src/composables/useStreamingTts.ts

• Step 1: Create the composable

Create frontend/src/composables/useStreamingTts.ts with the full implementation:

import { ref, watch, computed } from 'vue'
import type { Ref, ComputedRef } from 'vue'
import { synthesiseSpeech } from '@/api/client'
import { useVoiceAudio } from '@/composables/useVoiceAudio'

/** Minimum stripped character count to bother synthesizing. */
const MIN_CHARS = 3

/** Matches sentence-terminal punctuation followed by whitespace or end-of-string. */
const SENTENCE_BOUNDARY = /[.!?]+(?=\s|$)/

function stripMarkdown(text: string): string {
  return text
    .replace(/```[\s\S]*?```/g, '')
    .replace(/`[^`]+`/g, (m) => m.slice(1, -1))
    .replace(/#{1,6}\s+/g, '')
    .replace(/\*\*([^*]+)\*\*/g, '$1')
    .replace(/\*([^*]+)\*/g, '$1')
    .replace(/\[([^\]]+)\]\([^)]+\)/g, '$1')
    .replace(/^\s*[-*+]\s+/gm, '')
    .replace(/\n{2,}/g, ' ')
    .trim()
}

/**
 * Extract completed sentences from `text` using SENTENCE_BOUNDARY.
 * Returns the sentences found and the unconsumed remainder.
 */
function extractSentences(text: string): { sentences: string[]; remainder: string } {
  const sentences: string[] = []
  let remaining = text
  let match: RegExpExecArray | null

  while ((match = SENTENCE_BOUNDARY.exec(remaining)) !== null) {
    const boundary = match.index + match[0].length
    const sentence = remaining.slice(0, boundary).trim()
    if (sentence) sentences.push(sentence)
    remaining = remaining.slice(boundary)
  }

  return { sentences, remainder: remaining }
}

export interface UseStreamingTtsOptions {
  streamingContent: Ref<string> | ComputedRef<string>
  streaming: Ref<boolean> | ComputedRef<boolean>
  enabled: Ref<boolean> | ComputedRef<boolean>
}

export interface UseStreamingTtsReturn {
  /** True while any synthesis request is in-flight or audio is playing. */
  speaking: ComputedRef<boolean>
  /** Cancel all in-flight synthesis/playback and clear the queue. */
  stop: () => void
}

export function useStreamingTts(options: UseStreamingTtsOptions): UseStreamingTtsReturn {
  const { streamingContent, streaming, enabled } = options
  const audio = useVoiceAudio()

  let sentenceBuffer = ''
  let lastSeenLength = 0
  let abortId = 0
  let playQueue: Promise<void> = Promise.resolve()
  const pendingCount = ref(0)

  const speaking = computed(() => pendingCount.value > 0 || audio.playing.value)

  function stop(): void {
    abortId++
    sentenceBuffer = ''
    lastSeenLength = 0
    playQueue = Promise.resolve()
    audio.stop()
    pendingCount.value = 0
  }

  async function enqueueSentence(sentence: string, myAbortId: number): Promise<void> {
    const stripped = stripMarkdown(sentence)
    if (stripped.length < MIN_CHARS) return

    pendingCount.value++
    let blob: Blob | null = null
    try {
      blob = await synthesiseSpeech(stripped)
    } catch (e) {
      console.warn('[StreamingTTS] Synthesis failed, retrying sentence', { sentence: stripped, error: e })
      try {
        blob = await synthesiseSpeech(stripped)
      } catch (e2) {
        console.warn('[StreamingTTS] Retry also failed, skipping sentence', { sentence: stripped, error: e2 })
      }
    } finally {
      pendingCount.value--
    }

    if (!blob) return

    // Capture blob for the closure — TS can't narrow after async gap
    const resolvedBlob = blob
    playQueue = playQueue.then(async () => {
      if (abortId !== myAbortId) return
      await audio.play(resolvedBlob)
    })
  }

  function dispatchBuffer(flush: boolean): void {
    if (!enabled.value) return
    const myAbortId = abortId
    const { sentences, remainder } = extractSentences(sentenceBuffer)
    sentenceBuffer = flush ? '' : remainder
    for (const sentence of sentences) {
      enqueueSentence(sentence, myAbortId)
    }
    if (flush && remainder.trim().length >= MIN_CHARS) {
      enqueueSentence(remainder.trim(), myAbortId)
    }
  }

  // Watch accumulating content — extract new characters since last check
  watch(streamingContent, (newContent) => {
    if (!enabled.value) return
    const delta = newContent.slice(lastSeenLength)
    lastSeenLength = newContent.length
    sentenceBuffer += delta
    dispatchBuffer(false)
  })

  // Watch streaming flag — stop on new message start, flush on end
  watch(streaming, (isStreaming) => {
    if (!enabled.value) return
    if (isStreaming) {
      // New message starting — cancel previous response's audio
      stop()
    } else {
      // Stream ended — flush any remaining fragment
      dispatchBuffer(true)
      lastSeenLength = 0
    }
  })

  return { speaking, stop }
}

• Step 2: TypeScript check

cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40

Expected: no errors mentioning useStreamingTts.ts.

• Step 3: Commit

git add frontend/src/composables/useStreamingTts.ts
git commit -m "feat(tts): add useStreamingTts composable for sentence-level streaming"

---

Task 2: Update ChatView

Files:

• Modify: frontend/src/views/ChatView.vue

Current TTS code to remove (lines ~35–66):

// REMOVE these:
const synthesising = ref(false);

async function speakLastAssistantMessage() { ... }  // entire function

watch(() => store.streaming, async (streaming) => {
  if (!streaming && listenMode.value && voiceTtsEnabled.value) {
    await new Promise((r) => setTimeout(r, 200));
    await speakLastAssistantMessage();
  }
});

Also remove the synthesiseSpeech import from @/api/client (it is no longer called directly in this file).

• Step 1: Add import and replace TTS logic

In frontend/src/views/ChatView.vue:

1. Add to imports at the top of <script setup>:

import { useStreamingTts } from "@/composables/useStreamingTts";

2. Remove synthesiseSpeech from the @/api/client import line (keep other imports like apiGet, transcribeAudio).

3. Remove const synthesising = ref(false); (line ~35).

4. Remove the entire speakLastAssistantMessage function (lines ~38–59).

5. Remove the watch(() => store.streaming, ...) block that called speakLastAssistantMessage (lines ~61–66).

6. Add after const listenMode = useListenMode();:

const tts = useStreamingTts({
  streamingContent: computed(() => store.streamingContent),
  streaming: computed(() => store.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
});

• Step 2: Update template references

In the ChatView template, replace every occurrence of synthesising with tts.speaking.value:

Find (line ~919):

:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': synthesising || audio.playing.value }"

Replace with:

:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': tts.speaking.value }"

Find (line ~920):

@click="listenMode = !listenMode; if (listenMode) speakLastAssistantMessage()"

Replace with:

@click="listenMode = !listenMode; if (!listenMode) tts.stop()"

Find (line ~924):

<svg v-if="!synthesising && !audio.playing.value" ...>

Replace with:

<svg v-if="!tts.speaking.value" ...>

Note: the audio variable (useVoiceAudio()) is still used for the volume slider and PTT stop — do NOT remove it.

• Step 3: TypeScript check

cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40

Expected: no errors.

• Step 4: Commit

git add frontend/src/views/ChatView.vue
git commit -m "feat(tts): wire useStreamingTts into ChatView"

---

Task 3: Update BriefingView

Files:

• Modify: frontend/src/views/BriefingView.vue

Current TTS code to remove:

// REMOVE:
const synthesising = ref(false)

async function speakText(text: string) { ... }       // entire function
async function listenToLatest() { ... }              // entire function

// REMOVE this watch block (the TTS one — keep the other streaming watch):
watch(() => chatStore.streaming, async (streaming) => {
  if (!streaming && listenMode.value && voiceTtsEnabled.value) {
    await new Promise((r) => setTimeout(r, 200))
    await listenToLatest()
  }
})

Note: BriefingView has two watch(() => chatStore.streaming, ...) blocks. Keep the first one (lines ~152–156, which refreshes messages). Remove only the TTS one (lines ~327–332).

Also remove the synthesiseSpeech import from @/api/client.

• Step 1: Add import and replace TTS logic

In frontend/src/views/BriefingView.vue:

1. Add to imports:

import { useStreamingTts } from '@/composables/useStreamingTts'

2. Remove synthesiseSpeech from the @/api/client import line.

3. Remove const synthesising = ref(false).

4. Remove the entire speakText function.

5. Remove the entire listenToLatest function.

6. Remove the TTS watch(() => chatStore.streaming, ...) block (the one that calls listenToLatest).

7. Add after const listenMode = useListenMode():

const tts = useStreamingTts({
  streamingContent: computed(() => chatStore.streamingContent),
  streaming: computed(() => chatStore.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})

• Step 2: Update template references

Find the listen toggle button in the template. Replace synthesising references:

<!-- Before -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': synthesising || audio.playing.value }"
@click="listenMode ? (listenMode = false) : (listenMode = true, listenToLatest())"

<!-- After -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': tts.speaking.value }"
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"

Find the stop button:

<!-- Before -->
v-if="voiceTtsEnabled && (synthesising || audio.playing.value)"
@click="audio.stop(); synthesising = false"

<!-- After -->
v-if="voiceTtsEnabled && tts.speaking.value"
@click="tts.stop()"

Find the spinner SVG condition:

<!-- Before -->
<svg v-if="!synthesising && !audio.playing.value" ...>

<!-- After -->
<svg v-if="!tts.speaking.value" ...>

• Step 3: TypeScript check

cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40

Expected: no errors.

• Step 4: Commit

git add frontend/src/views/BriefingView.vue
git commit -m "feat(tts): wire useStreamingTts into BriefingView"

---

Task 4: Add streaming TTS to WorkspaceView

Files:

• Modify: frontend/src/views/WorkspaceView.vue

WorkspaceView has no TTS today. We add: listen mode toggle, useStreamingTts, and the listen button in the chat input toolbar.

• Step 1: Add imports and composable

In frontend/src/views/WorkspaceView.vue, add to the import block at the top of <script setup>:

import { useListenMode } from '@/composables/useListenMode'
import { useStreamingTts } from '@/composables/useStreamingTts'
import { useVoiceAudio } from '@/composables/useVoiceAudio'

After the existing store setup code (after const settingsStore = useSettingsStore()), add:

const listenMode = useListenMode()
const voiceTtsEnabled = computed(() => settingsStore.voiceTtsReady)
const audio = useVoiceAudio()
const tts = useStreamingTts({
  streamingContent: computed(() => chatStore.streamingContent),
  streaming: computed(() => chatStore.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})

• Step 2: Add listen mode button to template

In the <div class="chat-input-area"> section (around line 365), add the listen button before the abort/send button:

<div class="chat-input-area">
  <textarea
    ref="inputEl"
    v-model="messageInput"
    class="chat-input"
    :placeholder="chatStore.streaming ? 'Type to queue next message… (Enter to queue)' : 'Message the agent… (Enter to send)'"
    rows="1"
    @keydown="onInputKeydown"
    @input="autoResize"
  ></textarea>
  <!-- Listen mode toggle (TTS) -->
  <button
    v-if="voiceTtsEnabled"
    class="btn-listen-ws"
    :class="{ 'btn-listen-ws--active': listenMode, 'btn-listen-ws--busy': tts.speaking.value }"
    :title="listenMode ? 'Stop auto-read' : 'Read responses aloud'"
    aria-label="Toggle listen mode"
    @click="listenMode = !listenMode; if (!listenMode) tts.stop()"
  >
    <svg v-if="!tts.speaking.value" width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
      <path d="M3 9v6h4l5 5V4L7 9H3zm13.5 3c0-1.77-1.02-3.29-2.5-4.03v8.05c1.48-.73 2.5-2.25 2.5-4.02zM14 3.23v2.06c2.89.86 5 3.54 5 6.71s-2.11 5.85-5 6.71v2.06c4.01-.91 7-4.49 7-8.77s-2.99-7.86-7-8.77z"/>
    </svg>
    <svg v-else width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
      <path d="M18 12c0-1.77-1.02-3.29-2.5-4.03v2.21l2.45 2.45c.03-.2.05-.41.05-.63zm2.5 0c0 .94-.2 1.82-.54 2.64l1.51 1.51C21.8 14.82 22 13.43 22 12c0-4.28-2.99-7.86-7-8.77v2.06c2.89.86 5 3.54 5 6.71zM4.27 3L3 4.27 7.73 9H3v6h4l5 5v-6.73l4.25 4.25c-.67.52-1.42.93-2.25 1.18v2.06c1.38-.31 2.63-.95 3.69-1.81L19.73 21 21 19.73l-9-9L4.27 3zM12 4L9.91 6.09 12 8.18V4z"/>
    </svg>
  </button>
  <button
    v-if="chatStore.streaming"
    class="btn-abort"
    title="Stop generation"
    @click="chatStore.cancelGeneration()"
  >
    ■ Stop
  </button>
  <button
    v-else
    class="btn-send"
    :disabled="!messageInput.trim()"
    @click="sendMessage"
  >
    Send
  </button>
</div>

• Step 3: Add CSS for the listen button

In the <style> block, add:

.btn-listen-ws {
  flex-shrink: 0;
  display: flex;
  align-items: center;
  justify-content: center;
  width: 2rem;
  height: 2rem;
  border: 1px solid var(--color-border);
  border-radius: var(--radius-md);
  background: transparent;
  color: var(--color-text-muted);
  cursor: pointer;
  transition: color 0.15s, background 0.15s, border-color 0.15s;
}
.btn-listen-ws:hover {
  color: var(--color-text);
  border-color: var(--color-primary);
}
.btn-listen-ws--active {
  color: var(--color-primary);
  border-color: var(--color-primary);
  background: color-mix(in srgb, var(--color-primary) 10%, transparent);
}
.btn-listen-ws--busy {
  color: var(--color-primary);
  animation: pulse 1.2s ease-in-out infinite;
}

• Step 4: TypeScript check

cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40

Expected: no errors.

• Step 5: Commit

git add frontend/src/views/WorkspaceView.vue
git commit -m "feat(tts): add streaming TTS listen mode to WorkspaceView"

---

Task 5: Final integration check and push

• Step 1: Full TypeScript check

cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1

Expected: zero errors.

• Step 2: Verify no dead imports remain

grep -n "synthesiseSpeech\|speakLastAssistantMessage\|speakText\|listenToLatest\|synthesising" \
  frontend/src/views/ChatView.vue \
  frontend/src/views/BriefingView.vue \
  frontend/src/views/WorkspaceView.vue

Expected: no matches (all replaced).

• Step 3: Manual smoke test

1. Enable voice in Admin → Config
2. Open Chat, enable listen mode (speaker icon)
3. Send a message and watch: audio should begin playing the first sentence while the LLM is still streaming the response
4. Send another message mid-playback — previous audio should stop immediately
5. Toggle listen mode off mid-response — audio stops, tts.stop() called
6. Repeat in /briefing and /workspace/:id

• Step 4: Push

git push origin dev

---

Self-Review

Spec coverage check:

• ✅ Starts playing during generation (sentence-level queue, fires on each boundary)
• ✅ Automatic when listen mode on (enabled computed = listenMode && voiceTtsEnabled)
• ✅ ChatView updated
• ✅ BriefingView updated
• ✅ WorkspaceView added
• ✅ One retry before skipping on failure
• ✅ Failures logged via console.warn with sentence text and error
• ✅ stop() on new message start (watch streaming → true)
• ✅ Flush remaining buffer on stream end (watch streaming → false)
• ✅ Fragments < 3 chars skipped
• ✅ abortId prevents stale playback after stop

Type consistency:

• tts.speaking is ComputedRef<boolean> — accessed as tts.speaking.value in templates ✅
• tts.stop() called consistently across all three views ✅
• useStreamingTts options match usage in all three call sites ✅
• audio variable kept in ChatView (used by volume slider) — not removed ✅