diff --git a/docs/superpowers/plans/2026-04-05-android-voice-io.md b/docs/superpowers/plans/2026-04-05-android-voice-io.md new file mode 100644 index 0000000..cbe02a8 --- /dev/null +++ b/docs/superpowers/plans/2026-04-05-android-voice-io.md @@ -0,0 +1,1511 @@ +# Android Voice I/O Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add server-side STT and streaming TTS to the Android app — mic button in Chat, Quick Capture bar, and Briefing. + +**Architecture:** A shared `VoiceNotifier` (Riverpod `Notifier`) owns the recording lifecycle, silence detection, and TTS sentence queue. Screens pass an `onTranscript` callback when entering voice mode; TTS is fed back sentence-by-sentence via `feedContent()` as the SSE response streams in. All audio I/O goes through the existing `/api/voice/*` endpoints — no backend changes needed. + +**Tech Stack:** Flutter/Dart, `record ^5.0.0` (WebM/Opus recording), `just_audio ^0.9.x` (WAV playback), `permission_handler ^11.x` (already in pubspec), `path_provider ^2.x` (already in pubspec), Riverpod 3, Dio. + +--- + +## File map + +| Action | Path | Responsibility | +|---|---|---| +| Create | `lib/data/api/voice_api.dart` | Dio calls: checkStatus, transcribe, synthesise | +| Create | `lib/data/repositories/voice_repository.dart` | Thin wrapper around VoiceApi | +| Modify | `lib/providers/api_client_provider.dart` | Add voiceApiProvider, voiceRepositoryProvider | +| Create | `lib/providers/voice_provider.dart` | VoiceState + VoiceNotifier (recording, silence, TTS) | +| Create | `lib/widgets/voice_mic_button.dart` | Animated mic button widget | +| Modify | `lib/screens/chat/chat_screen.dart` | Add mic button + voice banner + TTS feed | +| Modify | `lib/app.dart` | Add mic button to _QuickCaptureBar | +| Modify | `lib/screens/briefing/briefing_screen.dart` | Add mic button + voice banner + TTS feed | +| Modify | `pubspec.yaml` | Add record, just_audio | +| Modify | `android/app/src/main/AndroidManifest.xml` | RECORD_AUDIO permission | +| Modify | `test/widget_test.dart` | Unit tests for pure voice logic | + +--- + +## Task 1: Dependencies and permissions + +**Files:** +- Modify: `pubspec.yaml` +- Modify: `android/app/src/main/AndroidManifest.xml` + +- [ ] **Step 1: Add packages to pubspec.yaml** + +In `pubspec.yaml`, add under `dependencies:` (after `url_launcher`): + +```yaml + record: ^5.0.0 + just_audio: ^0.9.39 +``` + +- [ ] **Step 2: Add RECORD_AUDIO permission to AndroidManifest.xml** + +In `android/app/src/main/AndroidManifest.xml`, add after the existing `` line: + +```xml + +``` + +- [ ] **Step 3: Install packages** + +```bash +cd /home/bvandeusen/Nextcloud/Projects/fabled_app +flutter pub get +``` + +Expected: resolves `record 5.x` and `just_audio 0.9.x` without conflicts. + +- [ ] **Step 4: Verify no compile errors** + +```bash +flutter analyze +``` + +Expected: no new errors. + +- [ ] **Step 5: Commit** + +```bash +git add pubspec.yaml pubspec.lock android/app/src/main/AndroidManifest.xml +git commit -m "feat(voice): add record + just_audio dependencies, RECORD_AUDIO permission" +``` + +--- + +## Task 2: VoiceApi and VoiceStatus model + +**Files:** +- Create: `lib/data/api/voice_api.dart` +- Modify: `test/widget_test.dart` + +- [ ] **Step 1: Write failing tests for VoiceStatus.fromJson** + +Add to `test/widget_test.dart`: + +```dart +import 'package:fabled_app/data/api/voice_api.dart'; +``` + +Add a new group at the bottom of `main()`: + +```dart + group('VoiceStatus.fromJson', () { + test('parses enabled with stt and tts available', () { + final status = VoiceStatus.fromJson({ + 'enabled': true, + 'stt': true, + 'tts': true, + }); + expect(status.enabled, isTrue); + expect(status.stt, isTrue); + expect(status.tts, isTrue); + }); + + test('parses disabled state', () { + final status = VoiceStatus.fromJson({ + 'enabled': false, + 'stt': false, + 'tts': false, + }); + expect(status.enabled, isFalse); + }); + + test('fully unavailable when enabled but stt or tts false', () { + final status = VoiceStatus.fromJson({ + 'enabled': true, + 'stt': false, + 'tts': true, + }); + expect(status.fullyAvailable, isFalse); + }); + }); +``` + +- [ ] **Step 2: Run test to confirm it fails** + +```bash +flutter test test/widget_test.dart +``` + +Expected: FAIL — `voice_api.dart` not found. + +- [ ] **Step 3: Create lib/data/api/voice_api.dart** + +```dart +import 'dart:typed_data'; + +import 'package:dio/dio.dart'; + +import 'api_client.dart'; + +class VoiceStatus { + final bool enabled; + final bool stt; + final bool tts; + + const VoiceStatus({ + required this.enabled, + required this.stt, + required this.tts, + }); + + /// True only when voice is enabled AND both STT and TTS are ready. + bool get fullyAvailable => enabled && stt && tts; + + factory VoiceStatus.fromJson(Map json) => VoiceStatus( + enabled: json['enabled'] as bool? ?? false, + stt: json['stt'] as bool? ?? false, + tts: json['tts'] as bool? ?? false, + ); +} + +class VoiceApi { + final Dio _dio; + const VoiceApi(this._dio); + + /// Check whether voice features are available on this server. + Future checkStatus() async { + try { + final response = await _dio.get('/api/voice/status'); + return VoiceStatus.fromJson(response.data as Map); + } on DioException catch (e) { + throw dioToApp(e); + } + } + + /// POST WebM/Opus audio bytes and return the transcript string. + /// Returns empty string on empty or error response. + Future transcribe(Uint8List audioBytes) async { + try { + final formData = FormData.fromMap({ + 'audio': MultipartFile.fromBytes( + audioBytes, + filename: 'audio.webm', + contentType: DioMediaType('audio', 'webm'), + ), + }); + final response = await _dio.post( + '/api/voice/transcribe', + data: formData, + options: Options( + receiveTimeout: const Duration(seconds: 60), + contentType: 'multipart/form-data', + ), + ); + final data = response.data as Map; + return (data['transcript'] as String? ?? '').trim(); + } on DioException catch (e) { + throw dioToApp(e); + } + } + + /// POST text and return raw WAV bytes. + Future synthesise(String text) async { + try { + final response = await _dio.post( + '/api/voice/synthesise', + data: {'text': text}, + options: Options(responseType: ResponseType.bytes), + ); + return Uint8List.fromList(response.data as List); + } on DioException catch (e) { + throw dioToApp(e); + } + } +} +``` + +- [ ] **Step 4: Run tests to confirm they pass** + +```bash +flutter test test/widget_test.dart +``` + +Expected: all tests pass including the new VoiceStatus group. + +- [ ] **Step 5: Commit** + +```bash +git add lib/data/api/voice_api.dart test/widget_test.dart +git commit -m "feat(voice): add VoiceApi and VoiceStatus model" +``` + +--- + +## Task 3: VoiceRepository and provider registration + +**Files:** +- Create: `lib/data/repositories/voice_repository.dart` +- Modify: `lib/providers/api_client_provider.dart` + +- [ ] **Step 1: Create lib/data/repositories/voice_repository.dart** + +```dart +import 'dart:typed_data'; + +import '../api/voice_api.dart'; + +class VoiceRepository { + final VoiceApi _api; + const VoiceRepository(this._api); + + Future checkStatus() => _api.checkStatus(); + Future transcribe(Uint8List audioBytes) => _api.transcribe(audioBytes); + Future synthesise(String text) => _api.synthesise(text); +} +``` + +- [ ] **Step 2: Register providers in lib/providers/api_client_provider.dart** + +Add the following imports at the top of the file (after the existing imports): + +```dart +import '../data/api/voice_api.dart'; +import '../data/repositories/voice_repository.dart'; +``` + +Add at the bottom of the file (after `settingsApiProvider`): + +```dart +final voiceApiProvider = Provider((ref) { + return VoiceApi(ref.watch(dioProvider)); +}); + +final voiceRepositoryProvider = Provider((ref) { + return VoiceRepository(ref.watch(voiceApiProvider)); +}); +``` + +- [ ] **Step 3: Verify no errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 4: Commit** + +```bash +git add lib/data/repositories/voice_repository.dart lib/providers/api_client_provider.dart +git commit -m "feat(voice): add VoiceRepository and provider registration" +``` + +--- + +## Task 4: VoiceNotifier — state, recording, silence detection, transcription + +**Files:** +- Create: `lib/providers/voice_provider.dart` +- Modify: `test/widget_test.dart` + +- [ ] **Step 1: Write failing tests for sentence extraction and markdown stripping** + +Add the following import to `test/widget_test.dart`: + +```dart +import 'package:fabled_app/providers/voice_provider.dart'; +``` + +Add a new group at the bottom of `main()`: + +```dart + group('VoiceNotifier sentence extraction', () { + test('extracts complete sentences at full stops', () { + final result = extractSentences('Hello world. How are you? I am fine!'); + expect(result.sentences, equals(['Hello world.', 'How are you?', 'I am fine!'])); + expect(result.remainder, equals('')); + }); + + test('leaves incomplete fragment in remainder', () { + final result = extractSentences('Hello world. Incomplete'); + expect(result.sentences, equals(['Hello world.'])); + expect(result.remainder, equals('Incomplete')); + }); + + test('returns empty sentences and full text when no boundary', () { + final result = extractSentences('No boundary here'); + expect(result.sentences, isEmpty); + expect(result.remainder, equals('No boundary here')); + }); + }); + + group('VoiceNotifier markdown stripping', () { + test('strips code fences', () { + expect(stripMarkdownForTts('Before\n```dart\ncode\n```\nAfter'), equals('Before After')); + }); + + test('strips bold and italic markers', () { + expect(stripMarkdownForTts('**bold** and *italic*'), equals('bold and italic')); + }); + + test('strips headers', () { + expect(stripMarkdownForTts('## Section title'), equals('Section title')); + }); + + test('keeps link text, removes URL', () { + expect(stripMarkdownForTts('[click here](https://example.com)'), equals('click here')); + }); + + test('strips list markers', () { + expect(stripMarkdownForTts('- item one\n- item two'), equals('item one item two')); + }); + }); +``` + +- [ ] **Step 2: Run test to confirm failures** + +```bash +flutter test test/widget_test.dart +``` + +Expected: FAIL — `voice_provider.dart` not found. + +- [ ] **Step 3: Create lib/providers/voice_provider.dart** + +```dart +import 'dart:async'; +import 'dart:collection'; +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:flutter_riverpod/flutter_riverpod.dart'; +import 'package:just_audio/just_audio.dart'; +import 'package:path_provider/path_provider.dart'; +import 'package:permission_handler/permission_handler.dart'; +import 'package:record/record.dart'; + +import 'api_client_provider.dart'; + +// ── Public helpers (also used by tests) ────────────────────────────────────── + +class SentenceResult { + final List sentences; + final String remainder; + const SentenceResult({required this.sentences, required this.remainder}); +} + +/// Extract completed sentences from [text] at `.`, `!`, `?` boundaries. +/// Returns the completed sentences and the unconsumed remainder. +SentenceResult extractSentences(String text) { + final boundary = RegExp(r'[.!?]+(?=\s|$)'); + final sentences = []; + var remaining = text; + RegExpMatch? match; + while ((match = boundary.firstMatch(remaining)) != null) { + final end = match!.end; + final sentence = remaining.substring(0, end).trim(); + if (sentence.isNotEmpty) sentences.add(sentence); + remaining = remaining.substring(end).trimLeft(); + } + return SentenceResult(sentences: sentences, remainder: remaining); +} + +/// Strip markdown formatting before sending text to TTS. +String stripMarkdownForTts(String text) { + return text + .replaceAll(RegExp(r'```[\s\S]*?```'), '') // fenced code blocks + .replaceAll(RegExp(r'`([^`]+)`'), r'$1') // inline code — keep content + .replaceAll(RegExp(r'#{1,6}\s+'), '') // headings + .replaceAll(RegExp(r'\*\*([^*]+)\*\*'), r'$1') // bold + .replaceAll(RegExp(r'\*([^*]+)\*'), r'$1') // italic + .replaceAll(RegExp(r'\[([^\]]+)\]\([^)]+\)'), r'$1') // links → text + .replaceAll(RegExp(r'^\s*[-*+]\s+', multiLine: true), '') // list markers + .replaceAll(RegExp(r'\n{2,}'), ' ') // multiple newlines → space + .replaceAll('\n', ' ') + .trim(); +} + +// ── State ───────────────────────────────────────────────────────────────────── + +enum VoiceMode { idle, recording, transcribing, playing } + +class VoiceState { + final VoiceMode mode; + final bool voiceModeActive; + final bool available; + + const VoiceState({ + this.mode = VoiceMode.idle, + this.voiceModeActive = false, + this.available = true, + }); + + VoiceState copyWith({ + VoiceMode? mode, + bool? voiceModeActive, + bool? available, + }) => + VoiceState( + mode: mode ?? this.mode, + voiceModeActive: voiceModeActive ?? this.voiceModeActive, + available: available ?? this.available, + ); +} + +// ── Provider ────────────────────────────────────────────────────────────────── + +final voiceProvider = + NotifierProvider(VoiceNotifier.new); + +// ── Notifier ────────────────────────────────────────────────────────────────── + +class VoiceNotifier extends Notifier { + // Audio I/O + AudioRecorder? _recorder; + AudioPlayer? _player; + StreamSubscription? _amplitudeSubscription; + + // Recording / silence detection + int _recordingStartMs = 0; + int _silenceMs = 0; + static const _silenceThresholdDb = -40.0; + static const _silenceDurationMs = 1500; + static const _minRecordingMs = 300; + String? _recordingPath; + + // Voice mode callbacks + Future Function(String transcript)? _onTranscript; + bool _enableTts = false; + + // Streaming TTS state + String _sentenceBuffer = ''; + int _lastSeenLength = 0; + bool _streamComplete = false; + + // TTS playback queue + final _ttsQueue = Queue(); + bool _ttsPlaying = false; + int _ttsCounter = 0; + Directory? _tempDir; + + @override + VoiceState build() { + _recorder = AudioRecorder(); + _player = AudioPlayer(); + ref.onDispose(() { + _amplitudeSubscription?.cancel(); + _recorder?.dispose(); + _player?.dispose(); + }); + return const VoiceState(); + } + + // ── Public API ────────────────────────────────────────────────────────────── + + /// Enter voice mode. Checks server availability and mic permission first. + /// [onTranscript] is called with the transcript when a recording completes. + /// [enableTts] — if true, TTS plays when [feedContent] is called. + Future enterVoiceMode({ + required Future Function(String transcript) onTranscript, + bool enableTts = false, + required void Function(String message) onError, + }) async { + if (state.voiceModeActive) { + exitVoiceMode(); + return; + } + + // Check server availability + try { + final status = + await ref.read(voiceRepositoryProvider).checkStatus(); + if (!status.fullyAvailable) { + onError('Voice not available on this server'); + return; + } + } catch (_) { + onError('Voice not available on this server'); + return; + } + + // Check microphone permission + final permStatus = await Permission.microphone.request(); + if (permStatus == PermissionStatus.denied || + permStatus == PermissionStatus.permanentlyDenied) { + onError('Microphone permission required'); + if (permStatus == PermissionStatus.permanentlyDenied) { + await openAppSettings(); + } + return; + } + + _onTranscript = onTranscript; + _enableTts = enableTts; + _tempDir = await getTemporaryDirectory(); + + state = state.copyWith(voiceModeActive: true, available: true); + await _startListening(); + } + + /// Exit voice mode, stop all recording and TTS. + void exitVoiceMode() { + _amplitudeSubscription?.cancel(); + _amplitudeSubscription = null; + _recorder?.stop(); + _player?.stop(); + _ttsQueue.clear(); + _ttsPlaying = false; + _sentenceBuffer = ''; + _lastSeenLength = 0; + _streamComplete = false; + _onTranscript = null; + state = const VoiceState(); + } + + /// Feed streaming assistant content for TTS synthesis. + /// Call this from screens with each updated [fullContent] string. + /// Set [isComplete] to true when streaming has finished. + void feedContent(String fullContent, {required bool isComplete}) { + if (!state.voiceModeActive || !_enableTts) return; + + final delta = fullContent.length > _lastSeenLength + ? fullContent.substring(_lastSeenLength) + : ''; + _lastSeenLength = fullContent.length; + _sentenceBuffer += delta; + + _dispatchSentences(flush: isComplete); + + if (isComplete) { + _streamComplete = true; + _checkRestartListening(); + } + } + + // ── Internal recording ────────────────────────────────────────────────────── + + Future _startListening() async { + if (!state.voiceModeActive) return; + + _silenceMs = 0; + _recordingStartMs = DateTime.now().millisecondsSinceEpoch; + state = state.copyWith(mode: VoiceMode.recording); + + final dir = _tempDir ?? await getTemporaryDirectory(); + _recordingPath = '${dir.path}/voice_rec_${DateTime.now().millisecondsSinceEpoch}.webm'; + + await _recorder!.start( + const RecordConfig(encoder: AudioEncoder.opus, sampleRate: 16000), + path: _recordingPath!, + ); + + _amplitudeSubscription?.cancel(); + _amplitudeSubscription = _recorder! + .onAmplitudeChanged(const Duration(milliseconds: 200)) + .listen(_onAmplitude); + } + + void _onAmplitude(Amplitude event) { + if (!state.voiceModeActive) return; + + final elapsed = + DateTime.now().millisecondsSinceEpoch - _recordingStartMs; + if (elapsed < _minRecordingMs) return; + + if (event.current < _silenceThresholdDb) { + _silenceMs += 200; + if (_silenceMs >= _silenceDurationMs) { + _amplitudeSubscription?.cancel(); + _handleSilence(); + } + } else { + _silenceMs = 0; + } + } + + Future _handleSilence() async { + if (!state.voiceModeActive) return; + state = state.copyWith(mode: VoiceMode.transcribing); + + final path = await _recorder!.stop(); + if (path == null || !state.voiceModeActive) return; + + try { + final bytes = await File(path).readAsBytes(); + await File(path).delete().catchError((_) {}); + + if (!state.voiceModeActive) return; + + final transcript = + await ref.read(voiceRepositoryProvider).transcribe(bytes); + + if (!state.voiceModeActive) return; + + if (transcript.isEmpty) { + // Empty transcript — restart silently + await _startListening(); + return; + } + + // Reset TTS state for this new turn + _sentenceBuffer = ''; + _lastSeenLength = 0; + _streamComplete = false; + + if (_enableTts) { + state = state.copyWith(mode: VoiceMode.playing); + } + + await _onTranscript?.call(transcript); + + // If TTS is not enabled, loop immediately + if (!_enableTts && state.voiceModeActive) { + await _startListening(); + } + } catch (_) { + // Network/API error — exit voice mode + exitVoiceMode(); + } + } + + // ── Internal TTS ──────────────────────────────────────────────────────────── + + void _dispatchSentences({required bool flush}) { + final result = extractSentences(_sentenceBuffer); + _sentenceBuffer = flush ? '' : result.remainder; + + for (final sentence in result.sentences) { + _enqueueSentence(sentence); + } + if (flush && result.remainder.trim().length >= 3) { + _enqueueSentence(result.remainder.trim()); + } + } + + void _enqueueSentence(String sentence) { + final stripped = stripMarkdownForTts(sentence); + if (stripped.length < 3) return; + // Fire-and-forget synthesis into queue + _synthesiseSentence(stripped); + } + + Future _synthesiseSentence(String text) async { + try { + final wavBytes = + await ref.read(voiceRepositoryProvider).synthesise(text); + if (!state.voiceModeActive) return; + _ttsQueue.add(wavBytes); + if (!_ttsPlaying) _drainTtsQueue(); + } catch (_) { + // Skip failed sentence — TTS errors are non-fatal + } + } + + Future _drainTtsQueue() async { + if (_ttsPlaying) return; + _ttsPlaying = true; + final dir = _tempDir ?? await getTemporaryDirectory(); + + try { + while (_ttsQueue.isNotEmpty && state.voiceModeActive) { + final wavBytes = _ttsQueue.removeFirst(); + final path = '${dir.path}/tts_${_ttsCounter++}.wav'; + final file = File(path); + await file.writeAsBytes(wavBytes); + + try { + await _player!.setFilePath(path); + await _player!.play(); + await _player!.processingStateStream.firstWhere( + (s) => s == ProcessingState.completed || s == ProcessingState.idle, + ); + } finally { + await file.delete().catchError((_) {}); + } + } + } finally { + _ttsPlaying = false; + } + + _checkRestartListening(); + } + + void _checkRestartListening() { + if (_streamComplete && + _ttsQueue.isEmpty && + !_ttsPlaying && + state.voiceModeActive) { + _streamComplete = false; + _lastSeenLength = 0; + _sentenceBuffer = ''; + _startListening(); + } + } +} +``` + +- [ ] **Step 4: Run tests to confirm they pass** + +```bash +flutter test test/widget_test.dart +``` + +Expected: all tests pass. + +- [ ] **Step 5: Verify no analyzer errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 6: Commit** + +```bash +git add lib/providers/voice_provider.dart test/widget_test.dart +git commit -m "feat(voice): add VoiceNotifier with recording, silence detection, and streaming TTS" +``` + +--- + +## Task 5: VoiceMicButton widget + +**Files:** +- Create: `lib/widgets/voice_mic_button.dart` + +- [ ] **Step 1: Create lib/widgets/voice_mic_button.dart** + +```dart +import 'package:flutter/material.dart'; + +import '../providers/voice_provider.dart'; + +/// Animated mic button that reflects the current [VoiceMode]. +/// +/// - idle: muted background, mic_none icon +/// - recording: red with pulsing shadow ring +/// - transcribing: indigo with spinner +/// - playing: indigo with volume_up icon +class VoiceMicButton extends StatefulWidget { + final VoiceMode mode; + final bool voiceModeActive; + final VoidCallback? onTap; + + const VoiceMicButton({ + super.key, + required this.mode, + required this.voiceModeActive, + this.onTap, + }); + + @override + State createState() => _VoiceMicButtonState(); +} + +class _VoiceMicButtonState extends State + with SingleTickerProviderStateMixin { + late AnimationController _pulseController; + late Animation _pulseAnimation; + + @override + void initState() { + super.initState(); + _pulseController = AnimationController( + vsync: this, + duration: const Duration(milliseconds: 900), + )..repeat(reverse: true); + _pulseAnimation = Tween(begin: 1.0, end: 1.25).animate( + CurvedAnimation(parent: _pulseController, curve: Curves.easeInOut), + ); + } + + @override + void dispose() { + _pulseController.dispose(); + super.dispose(); + } + + Color _bgColor(BuildContext context) { + final cs = Theme.of(context).colorScheme; + return switch (widget.mode) { + VoiceMode.recording => const Color(0xFFEF4444), + VoiceMode.transcribing || VoiceMode.playing => cs.primary, + VoiceMode.idle => cs.surfaceContainerHighest, + }; + } + + Widget _icon(BuildContext context) { + final cs = Theme.of(context).colorScheme; + final iconColor = widget.mode == VoiceMode.idle + ? cs.onSurfaceVariant + : Colors.white; + + return switch (widget.mode) { + VoiceMode.transcribing => SizedBox( + width: 18, + height: 18, + child: CircularProgressIndicator( + strokeWidth: 2, + color: iconColor, + ), + ), + VoiceMode.playing => Icon(Icons.volume_up, color: iconColor, size: 20), + _ => Icon(Icons.mic, color: iconColor, size: 20), + }; + } + + @override + Widget build(BuildContext context) { + final isRecording = widget.mode == VoiceMode.recording; + + final button = Material( + color: _bgColor(context), + shape: const CircleBorder(), + child: InkWell( + customBorder: const CircleBorder(), + onTap: widget.onTap, + child: SizedBox( + width: 40, + height: 40, + child: Center(child: _icon(context)), + ), + ), + ); + + if (!isRecording) return button; + + return AnimatedBuilder( + animation: _pulseAnimation, + builder: (_, child) => Container( + decoration: BoxDecoration( + shape: BoxShape.circle, + boxShadow: [ + BoxShadow( + color: const Color(0xFFEF4444).withValues(alpha: 0.35), + blurRadius: 8 * _pulseAnimation.value, + spreadRadius: 2 * _pulseAnimation.value, + ), + ], + ), + child: child, + ), + child: button, + ); + } +} +``` + +- [ ] **Step 2: Verify no errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 3: Commit** + +```bash +git add lib/widgets/voice_mic_button.dart +git commit -m "feat(voice): add VoiceMicButton widget with animated states" +``` + +--- + +## Task 6: Chat screen integration + +**Files:** +- Modify: `lib/screens/chat/chat_screen.dart` + +The Chat screen needs: +1. A `VoiceMicButton` next to the send button +2. A red banner above the input row when voice mode is active +3. A `ref.listen` on `messagesProvider` that feeds content to `VoiceNotifier` during TTS + +- [ ] **Step 1: Add voice imports and mic button to chat_screen.dart** + +Replace the full contents of `lib/screens/chat/chat_screen.dart` with: + +```dart +import 'package:flutter/material.dart'; +import 'package:flutter_riverpod/flutter_riverpod.dart'; + +import '../../core/exceptions.dart'; +import '../../data/models/message.dart'; +import '../../providers/chat_provider.dart'; +import '../../providers/voice_provider.dart'; +import '../../widgets/chat_message_bubble.dart'; +import '../../widgets/voice_mic_button.dart'; + +class ChatScreen extends ConsumerStatefulWidget { + final int conversationId; + const ChatScreen({super.key, required this.conversationId}); + + @override + ConsumerState createState() => _ChatScreenState(); +} + +class _ChatScreenState extends ConsumerState { + final _controller = TextEditingController(); + final _scrollController = ScrollController(); + + @override + void dispose() { + _controller.dispose(); + _scrollController.dispose(); + // Exit voice mode if the user navigates away mid-session. + ref.read(voiceProvider.notifier).exitVoiceMode(); + super.dispose(); + } + + void _scrollToBottom() { + WidgetsBinding.instance.addPostFrameCallback((_) { + if (_scrollController.hasClients) { + _scrollController.animateTo( + _scrollController.position.maxScrollExtent, + duration: const Duration(milliseconds: 200), + curve: Curves.easeOut, + ); + } + }); + } + + Future _send() async { + final text = _controller.text.trim(); + if (text.isEmpty) return; + _controller.clear(); + try { + await ref + .read(messagesProvider(widget.conversationId).notifier) + .sendMessage(text); + } on AppException catch (e) { + if (mounted) { + ScaffoldMessenger.of(context) + .showSnackBar(SnackBar(content: Text(e.message))); + } + } catch (_) { + if (mounted) { + ScaffoldMessenger.of(context).showSnackBar( + const SnackBar(content: Text('Failed to send message.')), + ); + } + } + } + + Future _toggleVoiceMode() async { + final voice = ref.read(voiceProvider); + if (voice.voiceModeActive) { + ref.read(voiceProvider.notifier).exitVoiceMode(); + return; + } + await ref.read(voiceProvider.notifier).enterVoiceMode( + onTranscript: (transcript) async { + await ref + .read(messagesProvider(widget.conversationId).notifier) + .sendMessage(transcript); + }, + enableTts: true, + onError: (msg) { + if (mounted) { + ScaffoldMessenger.of(context) + .showSnackBar(SnackBar(content: Text(msg))); + } + }, + ); + } + + @override + Widget build(BuildContext context) { + final messagesAsync = ref.watch(messagesProvider(widget.conversationId)); + final isStreaming = ref.watch(isStreamingProvider(widget.conversationId)); + final voiceState = ref.watch(voiceProvider); + + // Scroll when messages change. + ref.listen(messagesProvider(widget.conversationId), (_, _) { + _scrollToBottom(); + }); + + // Feed streaming content to VoiceNotifier for TTS. + ref.listen(messagesProvider(widget.conversationId), (_, next) { + if (!voiceState.voiceModeActive) return; + final messages = next.value; + if (messages == null || messages.isEmpty) return; + final last = messages.last; + if (last.role != MessageRole.assistant) return; + final isComplete = last.status != 'generating'; + ref + .read(voiceProvider.notifier) + .feedContent(last.content, isComplete: isComplete); + }); + + final convTitle = ref + .watch(conversationsProvider) + .value + ?.where((c) => c.id == widget.conversationId) + .firstOrNull + ?.title; + + return Scaffold( + appBar: AppBar( + title: Text(convTitle?.isNotEmpty == true ? convTitle! : 'Chat'), + ), + body: Column( + children: [ + Expanded( + child: messagesAsync.when( + loading: () => + const Center(child: CircularProgressIndicator()), + error: (_, _) => const Center( + child: Text('Could not load messages.'), + ), + data: (messages) { + if (messages.isEmpty) { + return const Center( + child: Text('Send a message to start.')); + } + return ListView.builder( + controller: _scrollController, + padding: const EdgeInsets.symmetric( + horizontal: 8, vertical: 12), + itemCount: messages.length, + itemBuilder: (context, i) => + ChatMessageBubble(message: messages[i]), + ); + }, + ), + ), + // Voice mode banner + if (voiceState.voiceModeActive) + Container( + width: double.infinity, + color: const Color(0xFFEF4444).withValues(alpha: 0.12), + padding: + const EdgeInsets.symmetric(horizontal: 16, vertical: 6), + child: const Text( + '🎤 Listening… tap mic to exit voice mode', + style: TextStyle( + fontSize: 12, + color: Color(0xFFF87171), + ), + ), + ), + const Divider(height: 1), + SafeArea( + child: Padding( + padding: + const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Row( + children: [ + Expanded( + child: TextField( + controller: _controller, + decoration: InputDecoration( + hintText: voiceState.voiceModeActive + ? 'Listening…' + : 'Message…', + hintStyle: voiceState.voiceModeActive + ? const TextStyle(fontStyle: FontStyle.italic) + : null, + border: const OutlineInputBorder(), + isDense: true, + contentPadding: const EdgeInsets.symmetric( + horizontal: 12, vertical: 10), + ), + maxLines: + MediaQuery.of(context).size.width >= 600 ? 2 : 4, + minLines: 1, + textInputAction: TextInputAction.newline, + enabled: !isStreaming && !voiceState.voiceModeActive, + ), + ), + const SizedBox(width: 8), + VoiceMicButton( + mode: voiceState.mode, + voiceModeActive: voiceState.voiceModeActive, + onTap: _toggleVoiceMode, + ), + const SizedBox(width: 6), + IconButton.filled( + onPressed: (isStreaming || voiceState.voiceModeActive) + ? null + : _send, + icon: isStreaming + ? const SizedBox( + width: 20, + height: 20, + child: + CircularProgressIndicator(strokeWidth: 2), + ) + : const Icon(Icons.send), + ), + ], + ), + ), + ), + ], + ), + ); + } +} +``` + +- [ ] **Step 2: Verify no errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 3: Commit** + +```bash +git add lib/screens/chat/chat_screen.dart +git commit -m "feat(voice): integrate voice mode into Chat screen" +``` + +--- + +## Task 7: Quick Capture bar integration + +**Files:** +- Modify: `lib/app.dart` + +The `_QuickCaptureBarState` needs a `VoiceMicButton`. Recording is one-shot: transcript goes to `captureWorkQueueProvider.enqueue()`, no TTS, no loop. The mic button sits between the text field and the settings gear icon. + +- [ ] **Step 1: Add imports to lib/app.dart** + +At the top of `lib/app.dart`, add after the existing widget imports: + +```dart +import 'providers/voice_provider.dart'; +import 'widgets/voice_mic_button.dart'; +``` + +- [ ] **Step 2: Add _toggleCaptureMic method to _QuickCaptureBarState** + +In `_QuickCaptureBarState`, add this method after `_drainOfflineQueue`: + +```dart + Future _toggleCaptureMic() async { + final voice = ref.read(voiceProvider); + if (voice.voiceModeActive) { + ref.read(voiceProvider.notifier).exitVoiceMode(); + return; + } + await ref.read(voiceProvider.notifier).enterVoiceMode( + onTranscript: (transcript) async { + ref.read(captureWorkQueueProvider.notifier).enqueue(transcript); + }, + enableTts: false, + onError: (msg) { + if (!mounted) return; + ScaffoldMessenger.of(context) + .showSnackBar(SnackBar(content: Text(msg))); + }, + ); + } +``` + +- [ ] **Step 3: Add VoiceMicButton to the capture bar Row** + +In `_QuickCaptureBarState.build()`, locate the `Row` inside the `Padding` widget. The existing row has: `[Expanded(TextField), IconButton(settings)]`. + +Replace that `Row`'s `children` list with: + +```dart + children: [ + Expanded( + child: TextField( + controller: _controller, + textInputAction: TextInputAction.send, + onSubmitted: (_) => _submit(), + onChanged: (_) => setState(() {}), + decoration: InputDecoration( + hintText: ref.watch(voiceProvider).voiceModeActive + ? 'Listening…' + : _hintForLocation(location), + isDense: true, + contentPadding: const EdgeInsets.symmetric( + horizontal: 14, vertical: 10), + prefixIcon: totalPending > 0 + ? Badge( + label: Text('$totalPending'), + child: const Icon(Icons.cloud_upload_outlined), + ) + : isWorking + ? const Padding( + padding: EdgeInsets.all(12), + child: SizedBox( + width: 16, + height: 16, + child: CircularProgressIndicator( + strokeWidth: 2), + ), + ) + : const Icon(Icons.auto_awesome_outlined), + suffixIcon: _controller.text.trim().isNotEmpty + ? IconButton( + icon: const Icon(Icons.send), + onPressed: _submit, + tooltip: 'Capture', + ) + : null, + ), + ), + ), + VoiceMicButton( + mode: ref.watch(voiceProvider).mode, + voiceModeActive: ref.watch(voiceProvider).voiceModeActive, + onTap: _toggleCaptureMic, + ), + IconButton( + icon: const Icon(Icons.settings_outlined), + tooltip: 'Settings', + onPressed: () => context.push(Routes.settings), + ), + ], +``` + +- [ ] **Step 4: Verify no errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 5: Commit** + +```bash +git add lib/app.dart +git commit -m "feat(voice): integrate one-shot voice capture into Quick Capture bar" +``` + +--- + +## Task 8: Briefing screen integration + +**Files:** +- Modify: `lib/screens/briefing/briefing_screen.dart` + +Same pattern as Chat: mic button in the reply row, voice banner above input, and `ref.listen` on `briefingProvider` to feed TTS. + +- [ ] **Step 1: Add imports to briefing_screen.dart** + +At the top of `lib/screens/briefing/briefing_screen.dart`, add after existing imports: + +```dart +import '../../data/models/message.dart'; +import '../../providers/voice_provider.dart'; +import '../../widgets/voice_mic_button.dart'; +``` + +- [ ] **Step 2: Add exitVoiceMode to dispose** + +In `_BriefingScreenState.dispose()`, add before `super.dispose()`: + +```dart + ref.read(voiceProvider.notifier).exitVoiceMode(); +``` + +- [ ] **Step 3: Add _toggleVoiceMode method** + +In `_BriefingScreenState`, add after `_refresh`: + +```dart + Future _toggleVoiceMode() async { + final voice = ref.read(voiceProvider); + if (voice.voiceModeActive) { + ref.read(voiceProvider.notifier).exitVoiceMode(); + return; + } + await ref.read(voiceProvider.notifier).enterVoiceMode( + onTranscript: (transcript) async { + await ref.read(briefingProvider.notifier).sendReply(transcript); + }, + enableTts: true, + onError: (msg) { + if (mounted) { + ScaffoldMessenger.of(context) + .showSnackBar(SnackBar(content: Text(msg))); + } + }, + ); + } +``` + +- [ ] **Step 4: Add TTS feed listener in build** + +In `_BriefingScreenState.build()`, locate the existing `ref.listen(briefingProvider, ...)` call that scrolls to bottom. Add a second `ref.listen` immediately after it: + +```dart + // Feed streaming assistant content to VoiceNotifier for TTS. + ref.listen(briefingProvider, (_, next) { + final voiceState = ref.read(voiceProvider); + if (!voiceState.voiceModeActive) return; + final conv = next.value; + if (conv == null || conv.messages.isEmpty) return; + final last = conv.messages.last; + if (last.role != MessageRole.assistant) return; + final isComplete = last.status != 'generating'; + ref + .read(voiceProvider.notifier) + .feedContent(last.content, isComplete: isComplete); + }); +``` + +- [ ] **Step 5: Add voice banner and mic button to the reply row** + +Locate the reply bar section in `build()` — the `Column` inside `data: (conv)`. It currently ends with: + +```dart + // Reply bar + const Divider(height: 1), + SafeArea( + child: Padding( + padding: const EdgeInsets.fromLTRB(8, 6, 8, 6), + child: Row( + children: [ + Expanded( + child: TextField( ... ), + ), + const SizedBox(width: 8), + _GradientSendButton( + onPressed: isStreaming ? null : _sendReply, + isStreaming: isStreaming, + ), +``` + +Replace that section (from `// Reply bar` to the closing of its Column entry) with: + +```dart + // Voice mode banner + if (voiceState.voiceModeActive) + Container( + width: double.infinity, + color: const Color(0xFFEF4444).withValues(alpha: 0.12), + padding: const EdgeInsets.symmetric( + horizontal: 16, vertical: 6), + child: const Text( + '🎤 Listening… tap mic to exit voice mode', + style: TextStyle( + fontSize: 12, + color: Color(0xFFF87171), + ), + ), + ), + // Reply bar + const Divider(height: 1), + SafeArea( + child: Padding( + padding: const EdgeInsets.fromLTRB(8, 6, 8, 6), + child: Row( + children: [ + Expanded( + child: TextField( + controller: _controller, + decoration: InputDecoration( + hintText: voiceState.voiceModeActive + ? 'Listening…' + : 'Reply to your briefing…', + hintStyle: voiceState.voiceModeActive + ? const TextStyle(fontStyle: FontStyle.italic) + : null, + border: const OutlineInputBorder(), + isDense: true, + contentPadding: const EdgeInsets.symmetric( + horizontal: 12, vertical: 10), + ), + minLines: 1, + maxLines: 4, + textInputAction: TextInputAction.newline, + enabled: !isStreaming && !voiceState.voiceModeActive, + ), + ), + const SizedBox(width: 8), + VoiceMicButton( + mode: voiceState.mode, + voiceModeActive: voiceState.voiceModeActive, + onTap: _toggleVoiceMode, + ), + const SizedBox(width: 6), + _GradientSendButton( + onPressed: (isStreaming || voiceState.voiceModeActive) + ? null + : _sendReply, + isStreaming: isStreaming, + ), +``` + +- [ ] **Step 6: Add voiceState variable in build** + +At the top of `_BriefingScreenState.build()`, just after the existing `final isStreaming = ...` line, add: + +```dart + final voiceState = ref.watch(voiceProvider); +``` + +- [ ] **Step 7: Verify no errors** + +```bash +flutter analyze +``` + +Expected: no errors. + +- [ ] **Step 8: Commit** + +```bash +git add lib/screens/briefing/briefing_screen.dart +git commit -m "feat(voice): integrate voice mode into Briefing screen" +``` + +--- + +## Task 9: Final verification + +- [ ] **Step 1: Run all tests** + +```bash +cd /home/bvandeusen/Nextcloud/Projects/fabled_app +flutter test test/widget_test.dart +``` + +Expected: all tests pass. + +- [ ] **Step 2: Full static analysis** + +```bash +flutter analyze +``` + +Expected: no errors or warnings related to voice code. + +- [ ] **Step 3: Build debug APK to confirm it compiles** + +```bash +flutter build apk --debug 2>&1 | tail -20 +``` + +Expected: `✓ Built build/app/outputs/flutter-apk/app-debug.apk` + +- [ ] **Step 4: Push to dev** + +```bash +cd /home/bvandeusen/Nextcloud/Projects/fabled_app +git push origin dev +``` + +--- + +## Manual smoke test checklist + +After installing the debug APK on device: + +- [ ] Chat screen: tap mic → permission dialog appears on first use +- [ ] Chat screen: speak → silence → transcript appears and message sends +- [ ] Chat screen: assistant response audio plays sentence by sentence +- [ ] Chat screen: tap mic again → voice mode exits, input re-enables +- [ ] Chat screen: navigate away mid-session → no audio or recording continues +- [ ] Quick Capture: tap mic → speak → transcript captured as note/task, mic returns to idle +- [ ] Briefing screen: tap mic → speak → reply sent, TTS plays response +- [ ] Server with voice disabled: tap mic → snackbar "Voice not available on this server" +- [ ] Deny mic permission: snackbar "Microphone permission required"