# Android Voice I/O Design

## Goal

Add voice input (STT) and output (TTS) to the Android app. All audio processing runs server-side via existing backend endpoints — no on-device STT or TTS APIs. Voice input is available in three locations: the Chat screen, the Quick Capture bar, and the Briefing follow-up bar. TTS auto-plays when voice mode is active.

---

## Architecture

### New files

| File | Responsibility |
|---|---|
| `lib/data/api/voice_api.dart` | Dio wrapper: `checkStatus()`, `transcribe(Uint8List)`, `synthesise(String)` |
| `lib/data/repositories/voice_repository.dart` | Thin wrapper around `VoiceApi` |
| `lib/providers/voice_provider.dart` | `VoiceState` + `VoiceNotifier extends Notifier<VoiceState>` — owns full recording/TTS lifecycle |
| `lib/widgets/voice_mic_button.dart` | Shared mic button widget used by all three screens; animates across all states |

### Modified files

| File | Change |
|---|---|
| `pubspec.yaml` | Add `record: ^6.x`, `just_audio: ^0.9.x` |
| `android/app/src/main/AndroidManifest.xml` | Add `RECORD_AUDIO` permission |
| `lib/providers/api_client_provider.dart` | Add `voiceApiProvider`, `voiceRepositoryProvider` |
| `lib/screens/chat/chat_screen.dart` | Add `VoiceMicButton` to input bar; wire streaming TTS watcher |
| `lib/app.dart` (`_QuickCaptureBar`) | Add `VoiceMicButton` next to capture send button |
| `lib/screens/briefing/briefing_screen.dart` | Add `VoiceMicButton` to follow-up input row; wire streaming TTS watcher |

### Packages

- **`record` ^6.x** — records audio on Android, outputs WebM/Opus (matches what the backend Whisper STT expects)
- **`just_audio` ^0.9.x** — queue-based audio player; plays WAV bytes returned by `/api/voice/synthesise`

---

## State model

```dart
enum VoiceMode { idle, recording, transcribing, playing }

class VoiceState {
  final VoiceMode mode;
  final bool voiceModeActive;  // whether the voice loop is running
  final bool available;        // from /api/voice/status check
}
```

`VoiceNotifier` is a `Notifier<VoiceState>` (Riverpod 3). It owns the `AudioRecorder` and `AudioPlayer` instances.

---

## Backend endpoints (no changes needed)

All existing, no backend work required:

- `GET /api/voice/status` → `{enabled, stt, tts}` — checked before entering voice mode
- `POST /api/voice/transcribe` — `multipart/form-data`, field `audio` (WebM/Opus bytes) → `{transcript, duration_ms}`
- `POST /api/voice/synthesise` — `{"text": "..."}` → `audio/wav` bytes

---

## Data flow

### Chat / Briefing — continuous voice loop

1. User taps mic → `VoiceNotifier.enterVoiceMode()`
2. Call `GET /api/voice/status`; if unavailable → show snackbar, abort
3. Request `RECORD_AUDIO` permission (via `permission_handler`); if denied → snackbar, abort
4. Set `voiceModeActive = true`; input field disabled, send button dimmed, red banner shown
5. Start recording via `record` package; poll amplitude every 200ms
6. Silence detected (amplitude < −40 dB for 1500ms) → stop recording
7. `POST /api/voice/transcribe` with WebM bytes → transcript
8. If transcript is empty → restart listening from step 5 silently
9. Call `sendMessage(transcript)` on `messagesProvider` (identical path to typed text)
10. As the SSE response streams in, `VoiceNotifier` watches the streaming content:
    - Buffer incoming text; extract completed sentences at `.`, `!`, `?` boundaries
    - Strip markdown (code fences, headers, bold/italic markers) before synthesising
    - For each sentence: `POST /api/voice/synthesise` → enqueue WAV blob in `just_audio` player
11. After all audio plays → restart listening from step 5
12. User taps mic again → `VoiceNotifier.exitVoiceMode()` → stop recording, cancel pending TTS, reset state

### Quick Capture — one-shot, no TTS loop

Steps 1–8 identical. Then instead of `sendMessage`, the transcript is passed to `captureWorkQueueProvider.enqueue(transcript)` — identical to typing in the capture bar. Mic returns to idle. No TTS playback, no loop.

---

## Silence detection

The `record` package emits `onAmplitudeChanged` events. `VoiceNotifier` tracks consecutive below-threshold samples:

- Threshold: −40 dB
- Required duration: 1500ms of continuous silence
- Minimum recording length: 300ms (ignore silence before user has spoken)

---

## Streaming TTS (mirrors web app)

Matches the behaviour of `useStreamingTts` in the web frontend:

1. Watch `messagesProvider` for streaming assistant content
2. Accumulate new characters into a sentence buffer
3. On each sentence boundary → strip markdown → `POST /api/voice/synthesise` → enqueue WAV
4. `just_audio` plays queued WAVs sequentially
5. On new message start → cancel in-flight synthesis, clear queue, stop playback
6. On stream end → flush remaining buffer fragment if ≥ 3 characters

Markdown stripping removes: code blocks (`` ``` ``), inline code, `#` headers, `**bold**`, `*italic*`, `[link](url)` → link text only, leading list markers.

---

## UX

### VoiceMicButton states

| State | Visual |
|---|---|
| Idle (voice mode off) | Plain mic icon, muted background |
| Recording | Red filled circle, pulsing shadow ring |
| Transcribing | Indigo filled circle, spinner overlay |
| Playing TTS | Indigo filled circle, speaker wave icon |

### Voice mode banner (Chat + Briefing only)

A thin red banner appears above the input row while voice mode is active:
> "🎤 Listening… tap mic to exit voice mode"

Input field shows "Listening…" hint in italic. Send button dimmed but still tappable as an override.

### Quick Capture bar

No banner. The capture bar's background shifts to a subtle red tint while recording. Returns to normal after capture.

---

## Error handling

| Scenario | Behaviour |
|---|---|
| Voice unavailable on server | Snackbar "Voice not available on this server", mic stays idle |
| Mic permission denied | Snackbar "Microphone permission required", mic stays idle |
| Empty transcript | Stay in voice mode, restart listening silently |
| Transcription API error | Stay in voice mode, restart listening; log warning |
| TTS synthesis failure for a sentence | Skip that sentence, continue playback queue |
| Network error during voice loop | Exit voice mode, show snackbar "Voice error — check connection" |
| User navigates away while in voice mode | `VoiceNotifier` disposes recording + playback cleanly |

---

## Permissions

Add to `android/app/src/main/AndroidManifest.xml`:

```xml
<uses-permission android:name="android.permission.RECORD_AUDIO"/>
```

Runtime permission requested via `permission_handler` at first mic tap. If permanently denied, open app settings.

---

## Testing

- **Unit — `VoiceNotifier`:** Mock `VoiceRepository`; verify state transitions: idle → recording → transcribing → idle (on empty transcript), idle → recording → transcribing → playing → recording (happy path)
- **Unit — silence detection:** Feed synthetic amplitude stream; assert stop fires after 1500ms below threshold, not before
- **Unit — streaming TTS sentence extraction:** Feed streaming content strings; assert correct sentences extracted, markdown stripped
- **Widget — `VoiceMicButton`:** Verify correct icon/colour/animation for each `VoiceMode` value
- **Integration — permission flow:** Mock `permission_handler`; assert denied path shows snackbar and stays idle

---

## Out of scope

- Wake word / always-on listening
- On-device STT or TTS
- Voice settings UI in the Android app (voice and TTS settings managed via the web Settings page)
- iOS support (Android only per project constraints)