# Streaming TTS Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Start playing TTS audio during LLM generation by splitting responses into sentences and synthesizing each sentence as it completes, rather than waiting for the full response.

**Architecture:** A new `useStreamingTts` composable watches `streamingContent` for sentence boundaries, fires per-sentence `synthesiseSpeech` requests concurrently, and plays audio in strict insertion order using `useVoiceAudio`. ChatView, BriefingView, and WorkspaceView all use this composable, replacing their current post-stream speak logic.

**Tech Stack:** Vue 3 Composition API, TypeScript, `useVoiceAudio` (existing), `synthesiseSpeech` from `api/client.ts` (existing), no backend changes.

---

## File Map

| Action | File | Responsibility |
|--------|------|----------------|
| **Create** | `frontend/src/composables/useStreamingTts.ts` | All streaming TTS logic: sentence splitting, TTS queuing, ordered playback |
| **Modify** | `frontend/src/views/ChatView.vue` | Replace `speakLastAssistantMessage` + old watch with `useStreamingTts` |
| **Modify** | `frontend/src/views/BriefingView.vue` | Replace `speakText` + `listenToLatest` + old watch with `useStreamingTts` |
| **Modify** | `frontend/src/views/WorkspaceView.vue` | Add listen mode toggle button + `useStreamingTts` |

---

## Task 1: Create `useStreamingTts` composable

**Files:**
- Create: `frontend/src/composables/useStreamingTts.ts`

- [ ] **Step 1: Create the composable**

Create `frontend/src/composables/useStreamingTts.ts` with the full implementation:

```typescript
import { ref, watch, computed } from 'vue'
import type { Ref, ComputedRef } from 'vue'
import { synthesiseSpeech } from '@/api/client'
import { useVoiceAudio } from '@/composables/useVoiceAudio'

/** Minimum stripped character count to bother synthesizing. */
const MIN_CHARS = 3

/** Matches sentence-terminal punctuation followed by whitespace or end-of-string. */
const SENTENCE_BOUNDARY = /[.!?]+(?=\s|$)/

function stripMarkdown(text: string): string {
  return text
    .replace(/```[\s\S]*?```/g, '')
    .replace(/`[^`]+`/g, (m) => m.slice(1, -1))
    .replace(/#{1,6}\s+/g, '')
    .replace(/\*\*([^*]+)\*\*/g, '$1')
    .replace(/\*([^*]+)\*/g, '$1')
    .replace(/\[([^\]]+)\]\([^)]+\)/g, '$1')
    .replace(/^\s*[-*+]\s+/gm, '')
    .replace(/\n{2,}/g, ' ')
    .trim()
}

/**
 * Extract completed sentences from `text` using SENTENCE_BOUNDARY.
 * Returns the sentences found and the unconsumed remainder.
 */
function extractSentences(text: string): { sentences: string[]; remainder: string } {
  const sentences: string[] = []
  let remaining = text
  let match: RegExpExecArray | null

  while ((match = SENTENCE_BOUNDARY.exec(remaining)) !== null) {
    const boundary = match.index + match[0].length
    const sentence = remaining.slice(0, boundary).trim()
    if (sentence) sentences.push(sentence)
    remaining = remaining.slice(boundary)
  }

  return { sentences, remainder: remaining }
}

export interface UseStreamingTtsOptions {
  streamingContent: Ref<string> | ComputedRef<string>
  streaming: Ref<boolean> | ComputedRef<boolean>
  enabled: Ref<boolean> | ComputedRef<boolean>
}

export interface UseStreamingTtsReturn {
  /** True while any synthesis request is in-flight or audio is playing. */
  speaking: ComputedRef<boolean>
  /** Cancel all in-flight synthesis/playback and clear the queue. */
  stop: () => void
}

export function useStreamingTts(options: UseStreamingTtsOptions): UseStreamingTtsReturn {
  const { streamingContent, streaming, enabled } = options
  const audio = useVoiceAudio()

  let sentenceBuffer = ''
  let lastSeenLength = 0
  let abortId = 0
  let playQueue: Promise<void> = Promise.resolve()
  const pendingCount = ref(0)

  const speaking = computed(() => pendingCount.value > 0 || audio.playing.value)

  function stop(): void {
    abortId++
    sentenceBuffer = ''
    lastSeenLength = 0
    playQueue = Promise.resolve()
    audio.stop()
    pendingCount.value = 0
  }

  async function enqueueSentence(sentence: string, myAbortId: number): Promise<void> {
    const stripped = stripMarkdown(sentence)
    if (stripped.length < MIN_CHARS) return

    pendingCount.value++
    let blob: Blob | null = null
    try {
      blob = await synthesiseSpeech(stripped)
    } catch (e) {
      console.warn('[StreamingTTS] Synthesis failed, retrying sentence', { sentence: stripped, error: e })
      try {
        blob = await synthesiseSpeech(stripped)
      } catch (e2) {
        console.warn('[StreamingTTS] Retry also failed, skipping sentence', { sentence: stripped, error: e2 })
      }
    } finally {
      pendingCount.value--
    }

    if (!blob) return

    // Capture blob for the closure — TS can't narrow after async gap
    const resolvedBlob = blob
    playQueue = playQueue.then(async () => {
      if (abortId !== myAbortId) return
      await audio.play(resolvedBlob)
    })
  }

  function dispatchBuffer(flush: boolean): void {
    if (!enabled.value) return
    const myAbortId = abortId
    const { sentences, remainder } = extractSentences(sentenceBuffer)
    sentenceBuffer = flush ? '' : remainder
    for (const sentence of sentences) {
      enqueueSentence(sentence, myAbortId)
    }
    if (flush && remainder.trim().length >= MIN_CHARS) {
      enqueueSentence(remainder.trim(), myAbortId)
    }
  }

  // Watch accumulating content — extract new characters since last check
  watch(streamingContent, (newContent) => {
    if (!enabled.value) return
    const delta = newContent.slice(lastSeenLength)
    lastSeenLength = newContent.length
    sentenceBuffer += delta
    dispatchBuffer(false)
  })

  // Watch streaming flag — stop on new message start, flush on end
  watch(streaming, (isStreaming) => {
    if (!enabled.value) return
    if (isStreaming) {
      // New message starting — cancel previous response's audio
      stop()
    } else {
      // Stream ended — flush any remaining fragment
      dispatchBuffer(true)
      lastSeenLength = 0
    }
  })

  return { speaking, stop }
}
```

- [ ] **Step 2: TypeScript check**

```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```

Expected: no errors mentioning `useStreamingTts.ts`.

- [ ] **Step 3: Commit**

```bash
git add frontend/src/composables/useStreamingTts.ts
git commit -m "feat(tts): add useStreamingTts composable for sentence-level streaming"
```

---

## Task 2: Update ChatView

**Files:**
- Modify: `frontend/src/views/ChatView.vue`

Current TTS code to remove (lines ~35–66):

```typescript
// REMOVE these:
const synthesising = ref(false);

async function speakLastAssistantMessage() { ... }  // entire function

watch(() => store.streaming, async (streaming) => {
  if (!streaming && listenMode.value && voiceTtsEnabled.value) {
    await new Promise((r) => setTimeout(r, 200));
    await speakLastAssistantMessage();
  }
});
```

Also remove the `synthesiseSpeech` import from `@/api/client` (it is no longer called directly in this file).

- [ ] **Step 1: Add import and replace TTS logic**

In `frontend/src/views/ChatView.vue`:

1. Add to imports at the top of `<script setup>`:
```typescript
import { useStreamingTts } from "@/composables/useStreamingTts";
```

2. Remove `synthesiseSpeech` from the `@/api/client` import line (keep other imports like `apiGet`, `transcribeAudio`).

3. Remove `const synthesising = ref(false);` (line ~35).

4. Remove the entire `speakLastAssistantMessage` function (lines ~38–59).

5. Remove the `watch(() => store.streaming, ...)` block that called `speakLastAssistantMessage` (lines ~61–66).

6. Add after `const listenMode = useListenMode();`:
```typescript
const tts = useStreamingTts({
  streamingContent: computed(() => store.streamingContent),
  streaming: computed(() => store.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
});
```

- [ ] **Step 2: Update template references**

In the ChatView template, replace every occurrence of `synthesising` with `tts.speaking.value`:

Find (line ~919):
```html
:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': synthesising || audio.playing.value }"
```
Replace with:
```html
:class="{ 'btn-listen--active': listenMode, 'btn-listen--busy': tts.speaking.value }"
```

Find (line ~920):
```html
@click="listenMode = !listenMode; if (listenMode) speakLastAssistantMessage()"
```
Replace with:
```html
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"
```

Find (line ~924):
```html
<svg v-if="!synthesising && !audio.playing.value" ...>
```
Replace with:
```html
<svg v-if="!tts.speaking.value" ...>
```

Note: the `audio` variable (`useVoiceAudio()`) is still used for the volume slider and PTT stop — do NOT remove it.

- [ ] **Step 3: TypeScript check**

```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```

Expected: no errors.

- [ ] **Step 4: Commit**

```bash
git add frontend/src/views/ChatView.vue
git commit -m "feat(tts): wire useStreamingTts into ChatView"
```

---

## Task 3: Update BriefingView

**Files:**
- Modify: `frontend/src/views/BriefingView.vue`

Current TTS code to remove:

```typescript
// REMOVE:
const synthesising = ref(false)

async function speakText(text: string) { ... }       // entire function
async function listenToLatest() { ... }              // entire function

// REMOVE this watch block (the TTS one — keep the other streaming watch):
watch(() => chatStore.streaming, async (streaming) => {
  if (!streaming && listenMode.value && voiceTtsEnabled.value) {
    await new Promise((r) => setTimeout(r, 200))
    await listenToLatest()
  }
})
```

Note: BriefingView has **two** `watch(() => chatStore.streaming, ...)` blocks. Keep the first one (lines ~152–156, which refreshes messages). Remove only the TTS one (lines ~327–332).

Also remove the `synthesiseSpeech` import from `@/api/client`.

- [ ] **Step 1: Add import and replace TTS logic**

In `frontend/src/views/BriefingView.vue`:

1. Add to imports:
```typescript
import { useStreamingTts } from '@/composables/useStreamingTts'
```

2. Remove `synthesiseSpeech` from the `@/api/client` import line.

3. Remove `const synthesising = ref(false)`.

4. Remove the entire `speakText` function.

5. Remove the entire `listenToLatest` function.

6. Remove the TTS `watch(() => chatStore.streaming, ...)` block (the one that calls `listenToLatest`).

7. Add after `const listenMode = useListenMode()`:
```typescript
const tts = useStreamingTts({
  streamingContent: computed(() => chatStore.streamingContent),
  streaming: computed(() => chatStore.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})
```

- [ ] **Step 2: Update template references**

Find the listen toggle button in the template. Replace `synthesising` references:

```html
<!-- Before -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': synthesising || audio.playing.value }"
@click="listenMode ? (listenMode = false) : (listenMode = true, listenToLatest())"

<!-- After -->
:class="{ 'btn-icon-active': listenMode, 'btn-icon-busy': tts.speaking.value }"
@click="listenMode = !listenMode; if (!listenMode) tts.stop()"
```

Find the stop button:
```html
<!-- Before -->
v-if="voiceTtsEnabled && (synthesising || audio.playing.value)"
@click="audio.stop(); synthesising = false"

<!-- After -->
v-if="voiceTtsEnabled && tts.speaking.value"
@click="tts.stop()"
```

Find the spinner SVG condition:
```html
<!-- Before -->
<svg v-if="!synthesising && !audio.playing.value" ...>

<!-- After -->
<svg v-if="!tts.speaking.value" ...>
```

- [ ] **Step 3: TypeScript check**

```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```

Expected: no errors.

- [ ] **Step 4: Commit**

```bash
git add frontend/src/views/BriefingView.vue
git commit -m "feat(tts): wire useStreamingTts into BriefingView"
```

---

## Task 4: Add streaming TTS to WorkspaceView

**Files:**
- Modify: `frontend/src/views/WorkspaceView.vue`

WorkspaceView has no TTS today. We add: listen mode toggle, `useStreamingTts`, and the listen button in the chat input toolbar.

- [ ] **Step 1: Add imports and composable**

In `frontend/src/views/WorkspaceView.vue`, add to the import block at the top of `<script setup>`:

```typescript
import { useListenMode } from '@/composables/useListenMode'
import { useStreamingTts } from '@/composables/useStreamingTts'
import { useVoiceAudio } from '@/composables/useVoiceAudio'
```

After the existing store setup code (after `const settingsStore = useSettingsStore()`), add:

```typescript
const listenMode = useListenMode()
const voiceTtsEnabled = computed(() => settingsStore.voiceTtsReady)
const audio = useVoiceAudio()
const tts = useStreamingTts({
  streamingContent: computed(() => chatStore.streamingContent),
  streaming: computed(() => chatStore.streaming),
  enabled: computed(() => listenMode.value && voiceTtsEnabled.value),
})
```

- [ ] **Step 2: Add listen mode button to template**

In the `<div class="chat-input-area">` section (around line 365), add the listen button before the abort/send button:

```html
<div class="chat-input-area">
  <textarea
    ref="inputEl"
    v-model="messageInput"
    class="chat-input"
    :placeholder="chatStore.streaming ? 'Type to queue next message… (Enter to queue)' : 'Message the agent… (Enter to send)'"
    rows="1"
    @keydown="onInputKeydown"
    @input="autoResize"
  ></textarea>
  <!-- Listen mode toggle (TTS) -->
  <button
    v-if="voiceTtsEnabled"
    class="btn-listen-ws"
    :class="{ 'btn-listen-ws--active': listenMode, 'btn-listen-ws--busy': tts.speaking.value }"
    :title="listenMode ? 'Stop auto-read' : 'Read responses aloud'"
    aria-label="Toggle listen mode"
    @click="listenMode = !listenMode; if (!listenMode) tts.stop()"
  >
    <svg v-if="!tts.speaking.value" width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
      <path d="M3 9v6h4l5 5V4L7 9H3zm13.5 3c0-1.77-1.02-3.29-2.5-4.03v8.05c1.48-.73 2.5-2.25 2.5-4.02zM14 3.23v2.06c2.89.86 5 3.54 5 6.71s-2.11 5.85-5 6.71v2.06c4.01-.91 7-4.49 7-8.77s-2.99-7.86-7-8.77z"/>
    </svg>
    <svg v-else width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
      <path d="M18 12c0-1.77-1.02-3.29-2.5-4.03v2.21l2.45 2.45c.03-.2.05-.41.05-.63zm2.5 0c0 .94-.2 1.82-.54 2.64l1.51 1.51C21.8 14.82 22 13.43 22 12c0-4.28-2.99-7.86-7-8.77v2.06c2.89.86 5 3.54 5 6.71zM4.27 3L3 4.27 7.73 9H3v6h4l5 5v-6.73l4.25 4.25c-.67.52-1.42.93-2.25 1.18v2.06c1.38-.31 2.63-.95 3.69-1.81L19.73 21 21 19.73l-9-9L4.27 3zM12 4L9.91 6.09 12 8.18V4z"/>
    </svg>
  </button>
  <button
    v-if="chatStore.streaming"
    class="btn-abort"
    title="Stop generation"
    @click="chatStore.cancelGeneration()"
  >
    ■ Stop
  </button>
  <button
    v-else
    class="btn-send"
    :disabled="!messageInput.trim()"
    @click="sendMessage"
  >
    Send
  </button>
</div>
```

- [ ] **Step 3: Add CSS for the listen button**

In the `<style>` block, add:

```css
.btn-listen-ws {
  flex-shrink: 0;
  display: flex;
  align-items: center;
  justify-content: center;
  width: 2rem;
  height: 2rem;
  border: 1px solid var(--color-border);
  border-radius: var(--radius-md);
  background: transparent;
  color: var(--color-text-muted);
  cursor: pointer;
  transition: color 0.15s, background 0.15s, border-color 0.15s;
}
.btn-listen-ws:hover {
  color: var(--color-text);
  border-color: var(--color-primary);
}
.btn-listen-ws--active {
  color: var(--color-primary);
  border-color: var(--color-primary);
  background: color-mix(in srgb, var(--color-primary) 10%, transparent);
}
.btn-listen-ws--busy {
  color: var(--color-primary);
  animation: pulse 1.2s ease-in-out infinite;
}
```

- [ ] **Step 4: TypeScript check**

```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1 | head -40
```

Expected: no errors.

- [ ] **Step 5: Commit**

```bash
git add frontend/src/views/WorkspaceView.vue
git commit -m "feat(tts): add streaming TTS listen mode to WorkspaceView"
```

---

## Task 5: Final integration check and push

- [ ] **Step 1: Full TypeScript check**

```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant/frontend
npx vue-tsc --noEmit 2>&1
```

Expected: zero errors.

- [ ] **Step 2: Verify no dead imports remain**

```bash
grep -n "synthesiseSpeech\|speakLastAssistantMessage\|speakText\|listenToLatest\|synthesising" \
  frontend/src/views/ChatView.vue \
  frontend/src/views/BriefingView.vue \
  frontend/src/views/WorkspaceView.vue
```

Expected: no matches (all replaced).

- [ ] **Step 3: Manual smoke test**

1. Enable voice in Admin → Config
2. Open Chat, enable listen mode (speaker icon)
3. Send a message and watch: audio should begin playing the first sentence while the LLM is still streaming the response
4. Send another message mid-playback — previous audio should stop immediately
5. Toggle listen mode off mid-response — audio stops, `tts.stop()` called
6. Repeat in `/briefing` and `/workspace/:id`

- [ ] **Step 4: Push**

```bash
git push origin dev
```

---

## Self-Review

**Spec coverage check:**
- ✅ Starts playing during generation (sentence-level queue, fires on each boundary)
- ✅ Automatic when listen mode on (enabled computed = listenMode && voiceTtsEnabled)
- ✅ ChatView updated
- ✅ BriefingView updated
- ✅ WorkspaceView added
- ✅ One retry before skipping on failure
- ✅ Failures logged via `console.warn` with sentence text and error
- ✅ `stop()` on new message start (watch streaming → true)
- ✅ Flush remaining buffer on stream end (watch streaming → false)
- ✅ Fragments < 3 chars skipped
- ✅ `abortId` prevents stale playback after stop

**Type consistency:**
- `tts.speaking` is `ComputedRef<boolean>` — accessed as `tts.speaking.value` in templates ✅
- `tts.stop()` called consistently across all three views ✅
- `useStreamingTts` options match usage in all three call sites ✅
- `audio` variable kept in ChatView (used by volume slider) — not removed ✅