docs: add web voice overlay polish implementation plan

2026-04-06 19:40:56 -04:00
parent 1b68559bfe
commit 9b69e38aff
1 changed files with 479 additions and 0 deletions
@@ -0,0 +1,479 @@
+# Web Voice Overlay Polish — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Ship the dormant `VoiceOverlay` component by mounting it in `App.vue`, wiring the Space bar shortcut, and replacing push-to-talk with click-to-toggle silence detection backed by a new `useSilenceDetector` composable.
+
+**Architecture:** A new `useSilenceDetector` composable uses `AudioContext` + `AnalyserNode` to monitor amplitude from a live `MediaStream` and fires a callback after sustained silence. `VoiceOverlay` coordinates recording and silence detection, switching from hold-to-record to click-to-toggle. `App.vue` mounts the overlay and adds a Space bar handler that dispatches the existing `voice:ptt-toggle` custom event.
+
+**Tech Stack:** Vue 3 Composition API, TypeScript, Web Audio API (`AudioContext`, `AnalyserNode`), existing `useVoiceRecorder` / `useVoiceAudio` composables.
+
+---
+
+## File Map
+
+| Action | Path |
+|--------|------|
+| Create | `frontend/src/composables/useSilenceDetector.ts` |
+| Modify | `frontend/src/composables/useVoiceRecorder.ts` |
+| Modify | `frontend/src/components/VoiceOverlay.vue` |
+| Modify | `frontend/src/App.vue` |
+
+---
+
+### Task 1: `useSilenceDetector` composable
+
+**Files:**
+- Create: `frontend/src/composables/useSilenceDetector.ts`
+
+**Context:** The Web Audio API lets us pipe a `MediaStream` into an `AnalyserNode` and read frequency data as a byte array every 100 ms. RMS amplitude of that array gives a 0–1 loudness value; converting to dB lets us use the same `-40 dB` threshold as the Android app. The composable must be safe to call `stop()` on multiple times and must reset amplitude to 0 after stopping so the animated bars collapse.
+
+- [ ] **Step 1: Create the file with full implementation**
+
+`frontend/src/composables/useSilenceDetector.ts`:
+
+```ts
+import { ref, readonly } from 'vue'
+
+export interface SilenceDetectorOptions {
+  thresholdDb?: number       // default -40
+  silenceDurationMs?: number // default 1500
+  minRecordingMs?: number    // default 500
+}
+
+export function useSilenceDetector(options: SilenceDetectorOptions = {}) {
+  const {
+    thresholdDb = -40,
+    silenceDurationMs = 1500,
+    minRecordingMs = 500,
+  } = options
+
+  const amplitude = ref(0)
+  let audioCtx: AudioContext | null = null
+  let intervalId: ReturnType<typeof setInterval> | null = null
+  let silenceMs = 0
+  let startedAt = 0
+
+  function start(stream: MediaStream, onSilence: () => void): void {
+    stop()
+    audioCtx = new AudioContext()
+    const source = audioCtx.createMediaStreamSource(stream)
+    const analyser = audioCtx.createAnalyser()
+    analyser.fftSize = 256
+    source.connect(analyser)
+
+    const data = new Uint8Array(analyser.frequencyBinCount)
+    silenceMs = 0
+    startedAt = Date.now()
+
+    intervalId = setInterval(() => {
+      analyser.getByteFrequencyData(data)
+      const rms = Math.sqrt(data.reduce((s, v) => s + v * v, 0) / data.length) / 255
+      amplitude.value = rms
+
+      const db = rms > 0 ? 20 * Math.log10(rms) : -100
+      if (db < thresholdDb) {
+        silenceMs += 100
+        if (silenceMs >= silenceDurationMs && Date.now() - startedAt >= minRecordingMs) {
+          stop()
+          onSilence()
+        }
+      } else {
+        silenceMs = 0
+      }
+    }, 100)
+  }
+
+  function stop(): void {
+    if (intervalId !== null) {
+      clearInterval(intervalId)
+      intervalId = null
+    }
+    if (audioCtx) {
+      audioCtx.close().catch(() => {})
+      audioCtx = null
+    }
+    amplitude.value = 0
+    silenceMs = 0
+  }
+
+  return { amplitude: readonly(amplitude), start, stop }
+}
+```
+
+- [ ] **Step 2: Verify TypeScript compiles**
+
+```bash
+cd /path/to/fabledassistant/frontend
+npx tsc --noEmit
+```
+
+Expected: no errors.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add frontend/src/composables/useSilenceDetector.ts
+git commit -m "feat: add useSilenceDetector composable with Web Audio API amplitude monitoring"
+```
+
+---
+
+### Task 2: Expose `stream` ref from `useVoiceRecorder`
+
+**Files:**
+- Modify: `frontend/src/composables/useVoiceRecorder.ts`
+
+**Context:** Currently `stream` is a plain `let` variable inside the closure. `VoiceOverlay` needs to pass the live `MediaStream` to `useSilenceDetector.start()` after recording begins. Exposing it as a readonly `Ref<MediaStream | null>` is the minimal change — no other callers are broken because they don't currently read `stream` from the return value.
+
+The current file is at `frontend/src/composables/useVoiceRecorder.ts`. Read it before editing — the key lines to change are:
+
+1. Top of function body: `let stream: MediaStream | null = null` → `const streamRef = ref<MediaStream | null>(null)`
+2. In `startRecording()`: `stream = await navigator.mediaDevices.getUserMedia({ audio: true })` → `streamRef.value = await navigator.mediaDevices.getUserMedia({ audio: true })`
+3. In `startRecording()` catch block: `stream = null` if present — replace with `streamRef.value = null` (if the catch sets stream to null; if not, skip)
+4. In `mediaRecorder.onstop`: `stream?.getTracks().forEach((t) => t.stop())` → `streamRef.value?.getTracks().forEach((t) => t.stop())` then `streamRef.value = null`
+5. Return object: add `stream: readonly(streamRef)`
+
+- [ ] **Step 1: Add the `ref` import if not already present**
+
+The file already imports `{ ref, readonly }` from `'vue'` — confirm this. If `ref` is missing from the import, add it.
+
+- [ ] **Step 2: Replace the `stream` variable declaration**
+
+Find:
+```ts
+let stream: MediaStream | null = null
+```
+Replace with:
+```ts
+const streamRef = ref<MediaStream | null>(null)
+```
+
+- [ ] **Step 3: Update all usages of `stream` in `startRecording`**
+
+Find:
+```ts
+      stream = await navigator.mediaDevices.getUserMedia({ audio: true })
+```
+Replace with:
+```ts
+      streamRef.value = await navigator.mediaDevices.getUserMedia({ audio: true })
+```
+
+- [ ] **Step 4: Update `onstop` handler**
+
+Find:
+```ts
+      stream?.getTracks().forEach((t) => t.stop())
+      stream = null
+```
+Replace with:
+```ts
+      streamRef.value?.getTracks().forEach((t) => t.stop())
+      streamRef.value = null
+```
+
+- [ ] **Step 5: Add `stream` to the return object**
+
+Find the return statement and add `stream: readonly(streamRef)`:
+```ts
+  return {
+    recording: readonly(recording),
+    error: readonly(error),
+    isSupported,
+    startRecording,
+    stopRecording,
+    stream: readonly(streamRef),
+  }
+```
+
+- [ ] **Step 6: Verify TypeScript compiles**
+
+```bash
+npx tsc --noEmit
+```
+
+Expected: no errors.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add frontend/src/composables/useVoiceRecorder.ts
+git commit -m "feat: expose live stream ref from useVoiceRecorder"
+```
+
+---
+
+### Task 3: Update `VoiceOverlay` — silence detection, click-to-toggle, amplitude bars
+
+**Files:**
+- Modify: `frontend/src/components/VoiceOverlay.vue`
+
+**Context:** `VoiceOverlay.vue` is a complete floating voice UI that was never mounted. It currently uses `@mousedown`/`@mouseup` for push-to-talk. This task switches it to click-to-toggle with automatic silence detection and adds animated amplitude bars during recording. Read the full file before making changes — the existing structure and style blocks must be preserved.
+
+#### Script changes
+
+- [ ] **Step 1: Import `useSilenceDetector`**
+
+At the top of `<script setup>`, after the existing imports, add:
+```ts
+import { useSilenceDetector } from '@/composables/useSilenceDetector'
+```
+
+- [ ] **Step 2: Instantiate the composable**
+
+After `const audio = useVoiceAudio()`, add:
+```ts
+const silenceDetector = useSilenceDetector()
+```
+
+- [ ] **Step 3: Update `startPtt` to start silence detection**
+
+Find the `startPtt` function. After `phase.value = 'recording'`, add:
+```ts
+  if (recorder.stream.value) {
+    silenceDetector.start(recorder.stream.value, stopPtt)
+  }
+```
+
+The complete `startPtt` after the change:
+```ts
+async function startPtt() {
+  if (!voiceEnabled.value || isBusy.value) return
+  audio.stop()
+  errorMsg.value = ''
+  open.value = true
+  await recorder.startRecording()
+  if (recorder.error.value) {
+    phase.value = 'error'
+    errorMsg.value = recorder.error.value
+    return
+  }
+  phase.value = 'recording'
+  if (recorder.stream.value) {
+    silenceDetector.start(recorder.stream.value, stopPtt)
+  }
+}
+```
+
+- [ ] **Step 4: Update `stopPtt` to stop silence detection**
+
+Add `silenceDetector.stop()` as the very first line of `stopPtt`:
+```ts
+async function stopPtt() {
+  silenceDetector.stop()
+  if (phase.value !== 'recording') return
+  // ... rest unchanged
+```
+
+- [ ] **Step 5: Update `cancelAll` to stop silence detection**
+
+Add `silenceDetector.stop()` after `recorder.stopRecording().catch(() => {})`:
+```ts
+function cancelAll() {
+  silenceDetector.stop()
+  recorder.stopRecording().catch(() => {})
+  audio.stop()
+  phase.value = 'idle'
+  streamContent.value = ''
+  errorMsg.value = ''
+}
+```
+
+- [ ] **Step 6: Add `onBtnClick` function**
+
+Add this function after `cancelAll`:
+```ts
+function onBtnClick() {
+  if (phase.value === 'error') { phase.value = 'idle'; return }
+  if (phase.value === 'recording') { stopPtt(); return }
+  if (phase.value === 'idle') { startPtt() }
+}
+```
+
+#### Template changes
+
+- [ ] **Step 7: Replace PTT mouse/touch handlers with `@click`**
+
+On `.voice-ptt-btn`, replace:
+```html
+@mousedown.prevent="startPtt"
+@mouseup.prevent="stopPtt"
+@touchstart.prevent="startPtt"
+@touchend.prevent="stopPtt"
+@click.prevent="phase === 'error' ? (phase = 'idle') : undefined"
+```
+with:
+```html
+@click.prevent="onBtnClick"
+```
+
+- [ ] **Step 8: Update aria-label and title on the button**
+
+Replace:
+```html
+:aria-label="phase === 'recording' ? 'Release to send' : 'Hold to speak'"
+:title="phase === 'recording' ? 'Release to send' : 'Hold Space or tap to speak'"
+```
+with:
+```html
+:aria-label="phase === 'recording' ? 'Click to stop' : 'Click to speak'"
+:title="phase === 'recording' ? 'Click to stop or wait for silence' : 'Click or press Space to speak'"
+```
+
+- [ ] **Step 9: Replace the static recording icon with amplitude bars**
+
+Find:
+```html
+          <!-- Recording: waveform / stop icon -->
+          <svg v-else-if="phase === 'recording'" width="22" height="22" viewBox="0 0 24 24" fill="currentColor">
+            <path d="M6 19h4V5H6v14zm8-14v14h4V5h-4z"/>
+          </svg>
+```
+Replace with:
+```html
+          <!-- Recording: amplitude bars -->
+          <span v-else-if="phase === 'recording'" class="voice-amp-bars">
+            <span
+              v-for="n in 3"
+              :key="n"
+              class="voice-amp-bar"
+              :style="{ transform: `scaleY(${0.3 + silenceDetector.amplitude.value * (0.4 + n * 0.15)})` }"
+            ></span>
+          </span>
+```
+
+- [ ] **Step 10: Update the idle hint label**
+
+Find:
+```html
+          Hold <kbd>Space</kbd> or tap
+```
+Replace with:
+```html
+          Tap or press <kbd>Space</kbd>
+```
+
+#### Style changes
+
+- [ ] **Step 11: Add amplitude bar styles to `<style scoped>`**
+
+Append inside the `<style scoped>` block:
+```css
+/* ─── Amplitude bars (recording state) ──────────────────────────────────── */
+.voice-amp-bars {
+  display: flex;
+  gap: 3px;
+  align-items: center;
+  height: 22px;
+}
+.voice-amp-bar {
+  width: 4px;
+  height: 18px;
+  background: #fff;
+  border-radius: 2px;
+  transform-origin: center;
+  transition: transform 0.08s ease;
+}
+```
+
+- [ ] **Step 12: Verify TypeScript compiles**
+
+```bash
+npx tsc --noEmit
+```
+
+Expected: no errors.
+
+- [ ] **Step 13: Commit**
+
+```bash
+git add frontend/src/components/VoiceOverlay.vue
+git commit -m "feat: click-to-toggle silence detection and amplitude bars in VoiceOverlay"
+```
+
+---
+
+### Task 4: Mount `VoiceOverlay` and wire Space bar in `App.vue`
+
+**Files:**
+- Modify: `frontend/src/App.vue`
+
+**Context:** `App.vue` has a full `onGlobalKeydown` handler and a shortcuts overlay. The Space bar is already documented there as "Hold to speak (voice, when enabled)" but the handler was never added to `onGlobalKeydown`. `VoiceOverlay` uses `Teleport to="body"` so it renders at the document root regardless of where it's placed in the template — just needs to be inside the authenticated block.
+
+#### Script changes
+
+- [ ] **Step 1: Add `VoiceOverlay` import**
+
+In `<script setup>`, after the existing component imports (after `ToastNotification`), add:
+```ts
+import VoiceOverlay from '@/components/VoiceOverlay.vue'
+```
+
+- [ ] **Step 2: Add Space bar case to `onGlobalKeydown`**
+
+The existing handler has a `switch (e.key)` block. The guard `if (isInputActive() || e.ctrlKey || e.metaKey || e.altKey) return` already runs before the switch, so the Space case only fires when the user isn't typing.
+
+Inside the `switch (e.key)` block, add this case after the existing `'c'` case:
+```ts
+    case ' ':
+      e.preventDefault()
+      document.dispatchEvent(new CustomEvent('voice:ptt-toggle'))
+      break
+```
+
+#### Template changes
+
+- [ ] **Step 3: Mount `VoiceOverlay` in the authenticated template**
+
+Find `<ToastNotification />` near the bottom of the authenticated template block and add `<VoiceOverlay />` directly above it:
+```html
+    <VoiceOverlay />
+    <ToastNotification />
+```
+
+- [ ] **Step 4: Update Space bar description in shortcuts panel**
+
+Find:
+```html
+                <span class="shortcut-desc">Hold to speak (voice, when enabled)</span>
+```
+Replace with:
+```html
+                <span class="shortcut-desc">Tap to speak (voice, when enabled)</span>
+```
+
+- [ ] **Step 5: Verify TypeScript compiles**
+
+```bash
+npx tsc --noEmit
+```
+
+Expected: no errors.
+
+- [ ] **Step 6: Verify full build succeeds**
+
+```bash
+npm run build
+```
+
+Expected: build completes with no errors.
+
+- [ ] **Step 7: Manual smoke test**
+
+1. Start the dev server: `npm run dev`
+2. Log in — confirm the floating mic button appears in the bottom-right corner
+3. Ensure voice is enabled in Settings → Voice
+4. Click the mic button — confirm it turns red with animated amplitude bars
+5. Speak — bars should animate with your voice
+6. Stop speaking — after ~1.5 s of silence, the button should switch to purple (transcribing), then green (speaking) as it plays back the response
+7. Click the mic while recording — confirm it stops immediately
+8. Press Space (not in an input field) — confirm it starts/stops recording
+9. Press Space in the chat input — confirm it does NOT trigger voice
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add frontend/src/App.vue
+git commit -m "feat: mount VoiceOverlay and wire Space bar shortcut in App.vue"
+```