FabledScribe/docs/superpowers/plans/2026-04-06-web-voice-overlay-polish.md

Web Voice Overlay Polish — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the dormant VoiceOverlay component by mounting it in App.vue, wiring the Space bar shortcut, and replacing push-to-talk with click-to-toggle silence detection backed by a new useSilenceDetector composable.

Architecture: A new useSilenceDetector composable uses AudioContext + AnalyserNode to monitor amplitude from a live MediaStream and fires a callback after sustained silence. VoiceOverlay coordinates recording and silence detection, switching from hold-to-record to click-to-toggle. App.vue mounts the overlay and adds a Space bar handler that dispatches the existing voice:ptt-toggle custom event.

Tech Stack: Vue 3 Composition API, TypeScript, Web Audio API (AudioContext, AnalyserNode), existing useVoiceRecorder / useVoiceAudio composables.

---

File Map

Action	Path
Create	frontend/src/composables/useSilenceDetector.ts
Modify	frontend/src/composables/useVoiceRecorder.ts
Modify	frontend/src/components/VoiceOverlay.vue
Modify	frontend/src/App.vue

---

Task 1: useSilenceDetector composable

Files:

• Create: frontend/src/composables/useSilenceDetector.ts

Context: The Web Audio API lets us pipe a MediaStream into an AnalyserNode and read frequency data as a byte array every 100 ms. RMS amplitude of that array gives a 0–1 loudness value; converting to dB lets us use the same -40 dB threshold as the Android app. The composable must be safe to call stop() on multiple times and must reset amplitude to 0 after stopping so the animated bars collapse.

• Step 1: Create the file with full implementation

frontend/src/composables/useSilenceDetector.ts:

import { ref, readonly } from 'vue'

export interface SilenceDetectorOptions {
  thresholdDb?: number       // default -40
  silenceDurationMs?: number // default 1500
  minRecordingMs?: number    // default 500
}

export function useSilenceDetector(options: SilenceDetectorOptions = {}) {
  const {
    thresholdDb = -40,
    silenceDurationMs = 1500,
    minRecordingMs = 500,
  } = options

  const amplitude = ref(0)
  let audioCtx: AudioContext | null = null
  let intervalId: ReturnType<typeof setInterval> | null = null
  let silenceMs = 0
  let startedAt = 0

  function start(stream: MediaStream, onSilence: () => void): void {
    stop()
    audioCtx = new AudioContext()
    const source = audioCtx.createMediaStreamSource(stream)
    const analyser = audioCtx.createAnalyser()
    analyser.fftSize = 256
    source.connect(analyser)

    const data = new Uint8Array(analyser.frequencyBinCount)
    silenceMs = 0
    startedAt = Date.now()

    intervalId = setInterval(() => {
      analyser.getByteFrequencyData(data)
      const rms = Math.sqrt(data.reduce((s, v) => s + v * v, 0) / data.length) / 255
      amplitude.value = rms

      const db = rms > 0 ? 20 * Math.log10(rms) : -100
      if (db < thresholdDb) {
        silenceMs += 100
        if (silenceMs >= silenceDurationMs && Date.now() - startedAt >= minRecordingMs) {
          stop()
          onSilence()
        }
      } else {
        silenceMs = 0
      }
    }, 100)
  }

  function stop(): void {
    if (intervalId !== null) {
      clearInterval(intervalId)
      intervalId = null
    }
    if (audioCtx) {
      audioCtx.close().catch(() => {})
      audioCtx = null
    }
    amplitude.value = 0
    silenceMs = 0
  }

  return { amplitude: readonly(amplitude), start, stop }
}

• Step 2: Verify TypeScript compiles

cd /path/to/fabledassistant/frontend
npx tsc --noEmit

Expected: no errors.

• Step 3: Commit

git add frontend/src/composables/useSilenceDetector.ts
git commit -m "feat: add useSilenceDetector composable with Web Audio API amplitude monitoring"

---

Task 2: Expose stream ref from useVoiceRecorder

Files:

• Modify: frontend/src/composables/useVoiceRecorder.ts

Context: Currently stream is a plain let variable inside the closure. VoiceOverlay needs to pass the live MediaStream to useSilenceDetector.start() after recording begins. Exposing it as a readonly Ref<MediaStream | null> is the minimal change — no other callers are broken because they don't currently read stream from the return value.

The current file is at frontend/src/composables/useVoiceRecorder.ts. Read it before editing — the key lines to change are:

1. Top of function body: let stream: MediaStream | null = null → const streamRef = ref<MediaStream | null>(null)
2. In startRecording(): stream = await navigator.mediaDevices.getUserMedia({ audio: true }) → streamRef.value = await navigator.mediaDevices.getUserMedia({ audio: true })
3. In startRecording() catch block: stream = null if present — replace with streamRef.value = null (if the catch sets stream to null; if not, skip)
4. In mediaRecorder.onstop: stream?.getTracks().forEach((t) => t.stop()) → streamRef.value?.getTracks().forEach((t) => t.stop()) then streamRef.value = null
5. Return object: add stream: readonly(streamRef)

• Step 1: Add the ref import if not already present

The file already imports { ref, readonly } from 'vue' — confirm this. If ref is missing from the import, add it.

• Step 2: Replace the stream variable declaration

Find:

let stream: MediaStream | null = null

Replace with:

const streamRef = ref<MediaStream | null>(null)

• Step 3: Update all usages of stream in startRecording

Find:

      stream = await navigator.mediaDevices.getUserMedia({ audio: true })

Replace with:

      streamRef.value = await navigator.mediaDevices.getUserMedia({ audio: true })

• Step 4: Update onstop handler

Find:

      stream?.getTracks().forEach((t) => t.stop())
      stream = null

Replace with:

      streamRef.value?.getTracks().forEach((t) => t.stop())
      streamRef.value = null

• Step 5: Add stream to the return object

Find the return statement and add stream: readonly(streamRef):

  return {
    recording: readonly(recording),
    error: readonly(error),
    isSupported,
    startRecording,
    stopRecording,
    stream: readonly(streamRef),
  }

• Step 6: Verify TypeScript compiles

npx tsc --noEmit

Expected: no errors.

• Step 7: Commit

git add frontend/src/composables/useVoiceRecorder.ts
git commit -m "feat: expose live stream ref from useVoiceRecorder"

---

Task 3: Update VoiceOverlay — silence detection, click-to-toggle, amplitude bars

Files:

• Modify: frontend/src/components/VoiceOverlay.vue

Context: VoiceOverlay.vue is a complete floating voice UI that was never mounted. It currently uses @mousedown/@mouseup for push-to-talk. This task switches it to click-to-toggle with automatic silence detection and adds animated amplitude bars during recording. Read the full file before making changes — the existing structure and style blocks must be preserved.

Script changes

• Step 1: Import useSilenceDetector

At the top of <script setup>, after the existing imports, add:

import { useSilenceDetector } from '@/composables/useSilenceDetector'

• Step 2: Instantiate the composable

After const audio = useVoiceAudio(), add:

const silenceDetector = useSilenceDetector()

• Step 3: Update startPtt to start silence detection

Find the startPtt function. After phase.value = 'recording', add:

  if (recorder.stream.value) {
    silenceDetector.start(recorder.stream.value, stopPtt)
  }

The complete startPtt after the change:

async function startPtt() {
  if (!voiceEnabled.value || isBusy.value) return
  audio.stop()
  errorMsg.value = ''
  open.value = true
  await recorder.startRecording()
  if (recorder.error.value) {
    phase.value = 'error'
    errorMsg.value = recorder.error.value
    return
  }
  phase.value = 'recording'
  if (recorder.stream.value) {
    silenceDetector.start(recorder.stream.value, stopPtt)
  }
}

• Step 4: Update stopPtt to stop silence detection

Add silenceDetector.stop() as the very first line of stopPtt:

async function stopPtt() {
  silenceDetector.stop()
  if (phase.value !== 'recording') return
  // ... rest unchanged

• Step 5: Update cancelAll to stop silence detection

Add silenceDetector.stop() after recorder.stopRecording().catch(() => {}):

function cancelAll() {
  silenceDetector.stop()
  recorder.stopRecording().catch(() => {})
  audio.stop()
  phase.value = 'idle'
  streamContent.value = ''
  errorMsg.value = ''
}

• Step 6: Add onBtnClick function

Add this function after cancelAll:

function onBtnClick() {
  if (phase.value === 'error') { phase.value = 'idle'; return }
  if (phase.value === 'recording') { stopPtt(); return }
  if (phase.value === 'idle') { startPtt() }
}

Template changes

• Step 7: Replace PTT mouse/touch handlers with @click

On .voice-ptt-btn, replace:

@mousedown.prevent="startPtt"
@mouseup.prevent="stopPtt"
@touchstart.prevent="startPtt"
@touchend.prevent="stopPtt"
@click.prevent="phase === 'error' ? (phase = 'idle') : undefined"

with:

@click.prevent="onBtnClick"

• Step 8: Update aria-label and title on the button

Replace:

:aria-label="phase === 'recording' ? 'Release to send' : 'Hold to speak'"
:title="phase === 'recording' ? 'Release to send' : 'Hold Space or tap to speak'"

with:

:aria-label="phase === 'recording' ? 'Click to stop' : 'Click to speak'"
:title="phase === 'recording' ? 'Click to stop or wait for silence' : 'Click or press Space to speak'"

• Step 9: Replace the static recording icon with amplitude bars

Find:

          <!-- Recording: waveform / stop icon -->
          <svg v-else-if="phase === 'recording'" width="22" height="22" viewBox="0 0 24 24" fill="currentColor">
            <path d="M6 19h4V5H6v14zm8-14v14h4V5h-4z"/>
          </svg>

Replace with:

          <!-- Recording: amplitude bars -->
          <span v-else-if="phase === 'recording'" class="voice-amp-bars">
            <span
              v-for="n in 3"
              :key="n"
              class="voice-amp-bar"
              :style="{ transform: `scaleY(${0.3 + silenceDetector.amplitude.value * (0.4 + n * 0.15)})` }"
            ></span>
          </span>

• Step 10: Update the idle hint label

Find:

          Hold <kbd>Space</kbd> or tap

Replace with:

          Tap or press <kbd>Space</kbd>

Style changes

• Step 11: Add amplitude bar styles to <style scoped>

Append inside the <style scoped> block:

/* ─── Amplitude bars (recording state) ──────────────────────────────────── */
.voice-amp-bars {
  display: flex;
  gap: 3px;
  align-items: center;
  height: 22px;
}
.voice-amp-bar {
  width: 4px;
  height: 18px;
  background: #fff;
  border-radius: 2px;
  transform-origin: center;
  transition: transform 0.08s ease;
}

• Step 12: Verify TypeScript compiles

npx tsc --noEmit

Expected: no errors.

• Step 13: Commit

git add frontend/src/components/VoiceOverlay.vue
git commit -m "feat: click-to-toggle silence detection and amplitude bars in VoiceOverlay"

---

Task 4: Mount VoiceOverlay and wire Space bar in App.vue

Files:

• Modify: frontend/src/App.vue

Context: App.vue has a full onGlobalKeydown handler and a shortcuts overlay. The Space bar is already documented there as "Hold to speak (voice, when enabled)" but the handler was never added to onGlobalKeydown. VoiceOverlay uses Teleport to="body" so it renders at the document root regardless of where it's placed in the template — just needs to be inside the authenticated block.

Script changes

• Step 1: Add VoiceOverlay import

In <script setup>, after the existing component imports (after ToastNotification), add:

import VoiceOverlay from '@/components/VoiceOverlay.vue'

• Step 2: Add Space bar case to onGlobalKeydown

The existing handler has a switch (e.key) block. The guard if (isInputActive() || e.ctrlKey || e.metaKey || e.altKey) return already runs before the switch, so the Space case only fires when the user isn't typing.

Inside the switch (e.key) block, add this case after the existing 'c' case:

    case ' ':
      e.preventDefault()
      document.dispatchEvent(new CustomEvent('voice:ptt-toggle'))
      break

Template changes

• Step 3: Mount VoiceOverlay in the authenticated template

Find <ToastNotification /> near the bottom of the authenticated template block and add <VoiceOverlay /> directly above it:

    <VoiceOverlay />
    <ToastNotification />

• Step 4: Update Space bar description in shortcuts panel

Find:

                <span class="shortcut-desc">Hold to speak (voice, when enabled)</span>

Replace with:

                <span class="shortcut-desc">Tap to speak (voice, when enabled)</span>

• Step 5: Verify TypeScript compiles

npx tsc --noEmit

Expected: no errors.

• Step 6: Verify full build succeeds

npm run build

Expected: build completes with no errors.

• Step 7: Manual smoke test

1. Start the dev server: npm run dev
2. Log in — confirm the floating mic button appears in the bottom-right corner
3. Ensure voice is enabled in Settings → Voice
4. Click the mic button — confirm it turns red with animated amplitude bars
5. Speak — bars should animate with your voice
6. Stop speaking — after ~1.5 s of silence, the button should switch to purple (transcribing), then green (speaking) as it plays back the response
7. Click the mic while recording — confirm it stops immediately
8. Press Space (not in an input field) — confirm it starts/stops recording
9. Press Space in the chat input — confirm it does NOT trigger voice

• Step 8: Commit

git add frontend/src/App.vue
git commit -m "feat: mount VoiceOverlay and wire Space bar shortcut in App.vue"