diff --git a/.gitignore b/.gitignore index f8554ec..46bdfa0 100644 --- a/.gitignore +++ b/.gitignore @@ -49,3 +49,4 @@ app.*.map.json /android/app/debug /android/app/profile /android/app/release +.superpowers/ diff --git a/docs/superpowers/specs/2026-04-05-android-voice-io-design.md b/docs/superpowers/specs/2026-04-05-android-voice-io-design.md new file mode 100644 index 0000000..1e5c8c7 --- /dev/null +++ b/docs/superpowers/specs/2026-04-05-android-voice-io-design.md @@ -0,0 +1,180 @@ +# Android Voice I/O Design + +## Goal + +Add voice input (STT) and output (TTS) to the Android app. All audio processing runs server-side via existing backend endpoints — no on-device STT or TTS APIs. Voice input is available in three locations: the Chat screen, the Quick Capture bar, and the Briefing follow-up bar. TTS auto-plays when voice mode is active. + +--- + +## Architecture + +### New files + +| File | Responsibility | +|---|---| +| `lib/data/api/voice_api.dart` | Dio wrapper: `checkStatus()`, `transcribe(Uint8List)`, `synthesise(String)` | +| `lib/data/repositories/voice_repository.dart` | Thin wrapper around `VoiceApi` | +| `lib/providers/voice_provider.dart` | `VoiceState` + `VoiceNotifier extends Notifier` — owns full recording/TTS lifecycle | +| `lib/widgets/voice_mic_button.dart` | Shared mic button widget used by all three screens; animates across all states | + +### Modified files + +| File | Change | +|---|---| +| `pubspec.yaml` | Add `record: ^6.x`, `just_audio: ^0.9.x` | +| `android/app/src/main/AndroidManifest.xml` | Add `RECORD_AUDIO` permission | +| `lib/providers/api_client_provider.dart` | Add `voiceApiProvider`, `voiceRepositoryProvider` | +| `lib/screens/chat/chat_screen.dart` | Add `VoiceMicButton` to input bar; wire streaming TTS watcher | +| `lib/app.dart` (`_QuickCaptureBar`) | Add `VoiceMicButton` next to capture send button | +| `lib/screens/briefing/briefing_screen.dart` | Add `VoiceMicButton` to follow-up input row; wire streaming TTS watcher | + +### Packages + +- **`record` ^6.x** — records audio on Android, outputs WebM/Opus (matches what the backend Whisper STT expects) +- **`just_audio` ^0.9.x** — queue-based audio player; plays WAV bytes returned by `/api/voice/synthesise` + +--- + +## State model + +```dart +enum VoiceMode { idle, recording, transcribing, playing } + +class VoiceState { + final VoiceMode mode; + final bool voiceModeActive; // whether the voice loop is running + final bool available; // from /api/voice/status check +} +``` + +`VoiceNotifier` is a `Notifier` (Riverpod 3). It owns the `AudioRecorder` and `AudioPlayer` instances. + +--- + +## Backend endpoints (no changes needed) + +All existing, no backend work required: + +- `GET /api/voice/status` → `{enabled, stt, tts}` — checked before entering voice mode +- `POST /api/voice/transcribe` — `multipart/form-data`, field `audio` (WebM/Opus bytes) → `{transcript, duration_ms}` +- `POST /api/voice/synthesise` — `{"text": "..."}` → `audio/wav` bytes + +--- + +## Data flow + +### Chat / Briefing — continuous voice loop + +1. User taps mic → `VoiceNotifier.enterVoiceMode()` +2. Call `GET /api/voice/status`; if unavailable → show snackbar, abort +3. Request `RECORD_AUDIO` permission (via `permission_handler`); if denied → snackbar, abort +4. Set `voiceModeActive = true`; input field disabled, send button dimmed, red banner shown +5. Start recording via `record` package; poll amplitude every 200ms +6. Silence detected (amplitude < −40 dB for 1500ms) → stop recording +7. `POST /api/voice/transcribe` with WebM bytes → transcript +8. If transcript is empty → restart listening from step 5 silently +9. Call `sendMessage(transcript)` on `messagesProvider` (identical path to typed text) +10. As the SSE response streams in, `VoiceNotifier` watches the streaming content: + - Buffer incoming text; extract completed sentences at `.`, `!`, `?` boundaries + - Strip markdown (code fences, headers, bold/italic markers) before synthesising + - For each sentence: `POST /api/voice/synthesise` → enqueue WAV blob in `just_audio` player +11. After all audio plays → restart listening from step 5 +12. User taps mic again → `VoiceNotifier.exitVoiceMode()` → stop recording, cancel pending TTS, reset state + +### Quick Capture — one-shot, no TTS loop + +Steps 1–8 identical. Then instead of `sendMessage`, the transcript is passed to `captureWorkQueueProvider.enqueue(transcript)` — identical to typing in the capture bar. Mic returns to idle. No TTS playback, no loop. + +--- + +## Silence detection + +The `record` package emits `onAmplitudeChanged` events. `VoiceNotifier` tracks consecutive below-threshold samples: + +- Threshold: −40 dB +- Required duration: 1500ms of continuous silence +- Minimum recording length: 300ms (ignore silence before user has spoken) + +--- + +## Streaming TTS (mirrors web app) + +Matches the behaviour of `useStreamingTts` in the web frontend: + +1. Watch `messagesProvider` for streaming assistant content +2. Accumulate new characters into a sentence buffer +3. On each sentence boundary → strip markdown → `POST /api/voice/synthesise` → enqueue WAV +4. `just_audio` plays queued WAVs sequentially +5. On new message start → cancel in-flight synthesis, clear queue, stop playback +6. On stream end → flush remaining buffer fragment if ≥ 3 characters + +Markdown stripping removes: code blocks (`` ``` ``), inline code, `#` headers, `**bold**`, `*italic*`, `[link](url)` → link text only, leading list markers. + +--- + +## UX + +### VoiceMicButton states + +| State | Visual | +|---|---| +| Idle (voice mode off) | Plain mic icon, muted background | +| Recording | Red filled circle, pulsing shadow ring | +| Transcribing | Indigo filled circle, spinner overlay | +| Playing TTS | Indigo filled circle, speaker wave icon | + +### Voice mode banner (Chat + Briefing only) + +A thin red banner appears above the input row while voice mode is active: +> "🎤 Listening… tap mic to exit voice mode" + +Input field shows "Listening…" hint in italic. Send button dimmed but still tappable as an override. + +### Quick Capture bar + +No banner. The capture bar's background shifts to a subtle red tint while recording. Returns to normal after capture. + +--- + +## Error handling + +| Scenario | Behaviour | +|---|---| +| Voice unavailable on server | Snackbar "Voice not available on this server", mic stays idle | +| Mic permission denied | Snackbar "Microphone permission required", mic stays idle | +| Empty transcript | Stay in voice mode, restart listening silently | +| Transcription API error | Stay in voice mode, restart listening; log warning | +| TTS synthesis failure for a sentence | Skip that sentence, continue playback queue | +| Network error during voice loop | Exit voice mode, show snackbar "Voice error — check connection" | +| User navigates away while in voice mode | `VoiceNotifier` disposes recording + playback cleanly | + +--- + +## Permissions + +Add to `android/app/src/main/AndroidManifest.xml`: + +```xml + +``` + +Runtime permission requested via `permission_handler` at first mic tap. If permanently denied, open app settings. + +--- + +## Testing + +- **Unit — `VoiceNotifier`:** Mock `VoiceRepository`; verify state transitions: idle → recording → transcribing → idle (on empty transcript), idle → recording → transcribing → playing → recording (happy path) +- **Unit — silence detection:** Feed synthetic amplitude stream; assert stop fires after 1500ms below threshold, not before +- **Unit — streaming TTS sentence extraction:** Feed streaming content strings; assert correct sentences extracted, markdown stripped +- **Widget — `VoiceMicButton`:** Verify correct icon/colour/animation for each `VoiceMode` value +- **Integration — permission flow:** Mock `permission_handler`; assert denied path shows snackbar and stays idle + +--- + +## Out of scope + +- Wake word / always-on listening +- On-device STT or TTS +- Voice settings UI in the Android app (voice and TTS settings managed via the web Settings page) +- iOS support (Android only per project constraints)