181 lines
7.4 KiB
Markdown
181 lines
7.4 KiB
Markdown
# Android Voice I/O Design
|
||
|
||
## Goal
|
||
|
||
Add voice input (STT) and output (TTS) to the Android app. All audio processing runs server-side via existing backend endpoints — no on-device STT or TTS APIs. Voice input is available in three locations: the Chat screen, the Quick Capture bar, and the Briefing follow-up bar. TTS auto-plays when voice mode is active.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### New files
|
||
|
||
| File | Responsibility |
|
||
|---|---|
|
||
| `lib/data/api/voice_api.dart` | Dio wrapper: `checkStatus()`, `transcribe(Uint8List)`, `synthesise(String)` |
|
||
| `lib/data/repositories/voice_repository.dart` | Thin wrapper around `VoiceApi` |
|
||
| `lib/providers/voice_provider.dart` | `VoiceState` + `VoiceNotifier extends Notifier<VoiceState>` — owns full recording/TTS lifecycle |
|
||
| `lib/widgets/voice_mic_button.dart` | Shared mic button widget used by all three screens; animates across all states |
|
||
|
||
### Modified files
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `pubspec.yaml` | Add `record: ^6.x`, `just_audio: ^0.9.x` |
|
||
| `android/app/src/main/AndroidManifest.xml` | Add `RECORD_AUDIO` permission |
|
||
| `lib/providers/api_client_provider.dart` | Add `voiceApiProvider`, `voiceRepositoryProvider` |
|
||
| `lib/screens/chat/chat_screen.dart` | Add `VoiceMicButton` to input bar; wire streaming TTS watcher |
|
||
| `lib/app.dart` (`_QuickCaptureBar`) | Add `VoiceMicButton` next to capture send button |
|
||
| `lib/screens/briefing/briefing_screen.dart` | Add `VoiceMicButton` to follow-up input row; wire streaming TTS watcher |
|
||
|
||
### Packages
|
||
|
||
- **`record` ^6.x** — records audio on Android, outputs WebM/Opus (matches what the backend Whisper STT expects)
|
||
- **`just_audio` ^0.9.x** — queue-based audio player; plays WAV bytes returned by `/api/voice/synthesise`
|
||
|
||
---
|
||
|
||
## State model
|
||
|
||
```dart
|
||
enum VoiceMode { idle, recording, transcribing, playing }
|
||
|
||
class VoiceState {
|
||
final VoiceMode mode;
|
||
final bool voiceModeActive; // whether the voice loop is running
|
||
final bool available; // from /api/voice/status check
|
||
}
|
||
```
|
||
|
||
`VoiceNotifier` is a `Notifier<VoiceState>` (Riverpod 3). It owns the `AudioRecorder` and `AudioPlayer` instances.
|
||
|
||
---
|
||
|
||
## Backend endpoints (no changes needed)
|
||
|
||
All existing, no backend work required:
|
||
|
||
- `GET /api/voice/status` → `{enabled, stt, tts}` — checked before entering voice mode
|
||
- `POST /api/voice/transcribe` — `multipart/form-data`, field `audio` (WebM/Opus bytes) → `{transcript, duration_ms}`
|
||
- `POST /api/voice/synthesise` — `{"text": "..."}` → `audio/wav` bytes
|
||
|
||
---
|
||
|
||
## Data flow
|
||
|
||
### Chat / Briefing — continuous voice loop
|
||
|
||
1. User taps mic → `VoiceNotifier.enterVoiceMode()`
|
||
2. Call `GET /api/voice/status`; if unavailable → show snackbar, abort
|
||
3. Request `RECORD_AUDIO` permission (via `permission_handler`); if denied → snackbar, abort
|
||
4. Set `voiceModeActive = true`; input field disabled, send button dimmed, red banner shown
|
||
5. Start recording via `record` package; poll amplitude every 200ms
|
||
6. Silence detected (amplitude < −40 dB for 1500ms) → stop recording
|
||
7. `POST /api/voice/transcribe` with WebM bytes → transcript
|
||
8. If transcript is empty → restart listening from step 5 silently
|
||
9. Call `sendMessage(transcript)` on `messagesProvider` (identical path to typed text)
|
||
10. As the SSE response streams in, `VoiceNotifier` watches the streaming content:
|
||
- Buffer incoming text; extract completed sentences at `.`, `!`, `?` boundaries
|
||
- Strip markdown (code fences, headers, bold/italic markers) before synthesising
|
||
- For each sentence: `POST /api/voice/synthesise` → enqueue WAV blob in `just_audio` player
|
||
11. After all audio plays → restart listening from step 5
|
||
12. User taps mic again → `VoiceNotifier.exitVoiceMode()` → stop recording, cancel pending TTS, reset state
|
||
|
||
### Quick Capture — one-shot, no TTS loop
|
||
|
||
Steps 1–8 identical. Then instead of `sendMessage`, the transcript is passed to `captureWorkQueueProvider.enqueue(transcript)` — identical to typing in the capture bar. Mic returns to idle. No TTS playback, no loop.
|
||
|
||
---
|
||
|
||
## Silence detection
|
||
|
||
The `record` package emits `onAmplitudeChanged` events. `VoiceNotifier` tracks consecutive below-threshold samples:
|
||
|
||
- Threshold: −40 dB
|
||
- Required duration: 1500ms of continuous silence
|
||
- Minimum recording length: 300ms (ignore silence before user has spoken)
|
||
|
||
---
|
||
|
||
## Streaming TTS (mirrors web app)
|
||
|
||
Matches the behaviour of `useStreamingTts` in the web frontend:
|
||
|
||
1. Watch `messagesProvider` for streaming assistant content
|
||
2. Accumulate new characters into a sentence buffer
|
||
3. On each sentence boundary → strip markdown → `POST /api/voice/synthesise` → enqueue WAV
|
||
4. `just_audio` plays queued WAVs sequentially
|
||
5. On new message start → cancel in-flight synthesis, clear queue, stop playback
|
||
6. On stream end → flush remaining buffer fragment if ≥ 3 characters
|
||
|
||
Markdown stripping removes: code blocks (`` ``` ``), inline code, `#` headers, `**bold**`, `*italic*`, `[link](url)` → link text only, leading list markers.
|
||
|
||
---
|
||
|
||
## UX
|
||
|
||
### VoiceMicButton states
|
||
|
||
| State | Visual |
|
||
|---|---|
|
||
| Idle (voice mode off) | Plain mic icon, muted background |
|
||
| Recording | Red filled circle, pulsing shadow ring |
|
||
| Transcribing | Indigo filled circle, spinner overlay |
|
||
| Playing TTS | Indigo filled circle, speaker wave icon |
|
||
|
||
### Voice mode banner (Chat + Briefing only)
|
||
|
||
A thin red banner appears above the input row while voice mode is active:
|
||
> "🎤 Listening… tap mic to exit voice mode"
|
||
|
||
Input field shows "Listening…" hint in italic. Send button dimmed but still tappable as an override.
|
||
|
||
### Quick Capture bar
|
||
|
||
No banner. The capture bar's background shifts to a subtle red tint while recording. Returns to normal after capture.
|
||
|
||
---
|
||
|
||
## Error handling
|
||
|
||
| Scenario | Behaviour |
|
||
|---|---|
|
||
| Voice unavailable on server | Snackbar "Voice not available on this server", mic stays idle |
|
||
| Mic permission denied | Snackbar "Microphone permission required", mic stays idle |
|
||
| Empty transcript | Stay in voice mode, restart listening silently |
|
||
| Transcription API error | Stay in voice mode, restart listening; log warning |
|
||
| TTS synthesis failure for a sentence | Skip that sentence, continue playback queue |
|
||
| Network error during voice loop | Exit voice mode, show snackbar "Voice error — check connection" |
|
||
| User navigates away while in voice mode | `VoiceNotifier` disposes recording + playback cleanly |
|
||
|
||
---
|
||
|
||
## Permissions
|
||
|
||
Add to `android/app/src/main/AndroidManifest.xml`:
|
||
|
||
```xml
|
||
<uses-permission android:name="android.permission.RECORD_AUDIO"/>
|
||
```
|
||
|
||
Runtime permission requested via `permission_handler` at first mic tap. If permanently denied, open app settings.
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
- **Unit — `VoiceNotifier`:** Mock `VoiceRepository`; verify state transitions: idle → recording → transcribing → idle (on empty transcript), idle → recording → transcribing → playing → recording (happy path)
|
||
- **Unit — silence detection:** Feed synthetic amplitude stream; assert stop fires after 1500ms below threshold, not before
|
||
- **Unit — streaming TTS sentence extraction:** Feed streaming content strings; assert correct sentences extracted, markdown stripped
|
||
- **Widget — `VoiceMicButton`:** Verify correct icon/colour/animation for each `VoiceMode` value
|
||
- **Integration — permission flow:** Mock `permission_handler`; assert denied path shows snackbar and stays idle
|
||
|
||
---
|
||
|
||
## Out of scope
|
||
|
||
- Wake word / always-on listening
|
||
- On-device STT or TTS
|
||
- Voice settings UI in the Android app (voice and TTS settings managed via the web Settings page)
|
||
- iOS support (Android only per project constraints)
|