FabledApp/docs/superpowers/specs/2026-04-05-android-voice-io-design.md

Android Voice I/O Design

Goal

Add voice input (STT) and output (TTS) to the Android app. All audio processing runs server-side via existing backend endpoints — no on-device STT or TTS APIs. Voice input is available in three locations: the Chat screen, the Quick Capture bar, and the Briefing follow-up bar. TTS auto-plays when voice mode is active.

---

Architecture

New files

File	Responsibility
lib/data/api/voice_api.dart	Dio wrapper: checkStatus(), transcribe(Uint8List), synthesise(String)
lib/data/repositories/voice_repository.dart	Thin wrapper around VoiceApi
lib/providers/voice_provider.dart	VoiceState + VoiceNotifier extends Notifier<VoiceState> — owns full recording/TTS lifecycle
lib/widgets/voice_mic_button.dart	Shared mic button widget used by all three screens; animates across all states

Modified files

File	Change
pubspec.yaml	Add record: ^6.x, just_audio: ^0.9.x
android/app/src/main/AndroidManifest.xml	Add RECORD_AUDIO permission
lib/providers/api_client_provider.dart	Add voiceApiProvider, voiceRepositoryProvider
lib/screens/chat/chat_screen.dart	Add VoiceMicButton to input bar; wire streaming TTS watcher
lib/app.dart (_QuickCaptureBar)	Add VoiceMicButton next to capture send button
lib/screens/briefing/briefing_screen.dart	Add VoiceMicButton to follow-up input row; wire streaming TTS watcher

Packages

• record ^6.x — records audio on Android, outputs WebM/Opus (matches what the backend Whisper STT expects)
• just_audio ^0.9.x — queue-based audio player; plays WAV bytes returned by /api/voice/synthesise

---

State model

enum VoiceMode { idle, recording, transcribing, playing }

class VoiceState {
  final VoiceMode mode;
  final bool voiceModeActive;  // whether the voice loop is running
  final bool available;        // from /api/voice/status check
}

VoiceNotifier is a Notifier<VoiceState> (Riverpod 3). It owns the AudioRecorder and AudioPlayer instances.

---

Backend endpoints (no changes needed)

All existing, no backend work required:

• GET /api/voice/status → {enabled, stt, tts} — checked before entering voice mode
• POST /api/voice/transcribe — multipart/form-data, field audio (WebM/Opus bytes) → {transcript, duration_ms}
• POST /api/voice/synthesise — {"text": "..."} → audio/wav bytes

---

Data flow

Chat / Briefing — continuous voice loop

1. User taps mic → VoiceNotifier.enterVoiceMode()
2. Call GET /api/voice/status; if unavailable → show snackbar, abort
3. Request RECORD_AUDIO permission (via permission_handler); if denied → snackbar, abort
4. Set voiceModeActive = true; input field disabled, send button dimmed, red banner shown
5. Start recording via record package; poll amplitude every 200ms
6. Silence detected (amplitude < −40 dB for 1500ms) → stop recording
7. POST /api/voice/transcribe with WebM bytes → transcript
8. If transcript is empty → restart listening from step 5 silently
9. Call sendMessage(transcript) on messagesProvider (identical path to typed text)
10. As the SSE response streams in, VoiceNotifier watches the streaming content:
    • Buffer incoming text; extract completed sentences at ., !, ? boundaries
    • Strip markdown (code fences, headers, bold/italic markers) before synthesising
    • For each sentence: POST /api/voice/synthesise → enqueue WAV blob in just_audio player
11. After all audio plays → restart listening from step 5
12. User taps mic again → VoiceNotifier.exitVoiceMode() → stop recording, cancel pending TTS, reset state

Quick Capture — one-shot, no TTS loop

Steps 1–8 identical. Then instead of sendMessage, the transcript is passed to captureWorkQueueProvider.enqueue(transcript) — identical to typing in the capture bar. Mic returns to idle. No TTS playback, no loop.

---

Silence detection

The record package emits onAmplitudeChanged events. VoiceNotifier tracks consecutive below-threshold samples:

• Threshold: −40 dB
• Required duration: 1500ms of continuous silence
• Minimum recording length: 300ms (ignore silence before user has spoken)

---

Streaming TTS (mirrors web app)

Matches the behaviour of useStreamingTts in the web frontend:

1. Watch messagesProvider for streaming assistant content
2. Accumulate new characters into a sentence buffer
3. On each sentence boundary → strip markdown → POST /api/voice/synthesise → enqueue WAV
4. just_audio plays queued WAVs sequentially
5. On new message start → cancel in-flight synthesis, clear queue, stop playback
6. On stream end → flush remaining buffer fragment if ≥ 3 characters

Markdown stripping removes: code blocks (```), inline code, # headers, **bold**, *italic*, [link](url) → link text only, leading list markers.

---

UX

VoiceMicButton states

State	Visual
Idle (voice mode off)	Plain mic icon, muted background
Recording	Red filled circle, pulsing shadow ring
Transcribing	Indigo filled circle, spinner overlay
Playing TTS	Indigo filled circle, speaker wave icon

Voice mode banner (Chat + Briefing only)

A thin red banner appears above the input row while voice mode is active:

"🎤 Listening… tap mic to exit voice mode"

Input field shows "Listening…" hint in italic. Send button dimmed but still tappable as an override.

Quick Capture bar

No banner. The capture bar's background shifts to a subtle red tint while recording. Returns to normal after capture.

---

Error handling

Scenario	Behaviour
Voice unavailable on server	Snackbar "Voice not available on this server", mic stays idle
Mic permission denied	Snackbar "Microphone permission required", mic stays idle
Empty transcript	Stay in voice mode, restart listening silently
Transcription API error	Stay in voice mode, restart listening; log warning
TTS synthesis failure for a sentence	Skip that sentence, continue playback queue
Network error during voice loop	Exit voice mode, show snackbar "Voice error — check connection"
User navigates away while in voice mode	VoiceNotifier disposes recording + playback cleanly

---

Permissions

Add to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO"/>

Runtime permission requested via permission_handler at first mic tap. If permanently denied, open app settings.

---

Testing

• Unit — VoiceNotifier: Mock VoiceRepository; verify state transitions: idle → recording → transcribing → idle (on empty transcript), idle → recording → transcribing → playing → recording (happy path)
• Unit — silence detection: Feed synthetic amplitude stream; assert stop fires after 1500ms below threshold, not before
• Unit — streaming TTS sentence extraction: Feed streaming content strings; assert correct sentences extracted, markdown stripped
• Widget — VoiceMicButton: Verify correct icon/colour/animation for each VoiceMode value
• Integration — permission flow: Mock permission_handler; assert denied path shows snackbar and stays idle

---

Out of scope

• Wake word / always-on listening
• On-device STT or TTS
• Voice settings UI in the Android app (voice and TTS settings managed via the web Settings page)
• iOS support (Android only per project constraints)