feat(voice): swap kokoro TTS → piper-tts
Kokoro has been stale upstream since April 2025 (`requires_python<3.13`), which broke the Python 3.14 build. Piper is the active replacement: maintained by OHF/Home Assistant, depends only on onnxruntime + pathvalidate (no torch, no spacy, no transformers), and has cp314 support today. Dockerfile: - Add `pip install piper-tts` after the STT install. - Bundle two default voices (en_US-amy-medium, en_US-ryan-medium) into /opt/piper-voices at build. Additional voices can be downloaded into /data/voices via the admin UI (separate commit). - Image add over the STT-only baseline: ~150 MB. services/tts.py — full rewrite: - New voice-discovery layer scans /opt/piper-voices + /data/voices for .onnx + .onnx.json pairs. /data wins over /opt for the same id so admin-downloaded voices can override bundled defaults. - Single PiperVoice kept warm; switches via _switch_voice() when the user changes their voice_tts_voice setting. - list_voices() returns metadata read from .onnx.json sidecars (label derived from filename, language, quality, sample_rate). - synthesise() uses piper's SynthesisConfig; converts kokoro-shaped `speed` multiplier to piper's `length_scale` (1.0 / speed). - `voice_blend` parameter accepted but ignored — piper has no blend equivalent; first entry's voice is used if anything is passed. - Dropped: HuggingFace commit-hash tracking (~80 lines), the daily check_for_kokoro_updates task, voice-tensor blending math. routes/voice.py: - tts_backend reports "piper" in /api/voice/status. - /api/voice/voices no longer requires tts_available() — even with the active voice failed to load, the catalog still lets the user pick a different one. - Synthesise request body dropped the voice_blend field; speed and voice still supported. alembic 0047_reset_voice_tts_settings: - Deletes any stored voice_tts_voice (kokoro IDs that don't map to piper) and voice_tts_blend (no piper equivalent) rows. Both re-default cleanly on next read. frontend: - VoiceBlendEntry type removed from api/client.ts. - synthesiseSpeech() signature dropped the voiceBlend parameter. - SettingsView.vue Voice Blend section removed entirely (slider, preview, slot management). voice_tts_blend save path removed. - Default voice id changed from "af_heart" to "en_US-amy-medium". - VoiceEntry gains optional language/quality/sample_rate fields from the richer piper sidecar metadata. Voice paths remain lazily guarded — `VOICE_ENABLED=false` (default) starts the app cleanly regardless of which TTS deps are present. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+20
-2
@@ -21,11 +21,29 @@ RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
# Python; the actual inference engine is ctranslate2 (C++) which has
|
||||
# cp314 wheels as of v4.7.2 (2026-05-19). No torch needed — ctranslate2
|
||||
# does its own CPU inference. Image add: ~150 MB.
|
||||
# TTS (piper-tts) is installed in a separate later step so STT can
|
||||
# stand alone if TTS becomes problematic.
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install faster-whisper soundfile
|
||||
|
||||
# Text-to-speech (piper-tts). Replaces kokoro, which has been
|
||||
# stale upstream since April 2025 (requires_python<3.13). Piper depends
|
||||
# only on onnxruntime (already pulled in for STT via faster-whisper) and
|
||||
# pathvalidate — total Python overhead is tiny. Voice models are separate
|
||||
# .onnx + .onnx.json files bundled below.
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install piper-tts
|
||||
|
||||
# Bundle two default voices in the image so first-run TTS works offline.
|
||||
# Additional voices can be downloaded at runtime into /data/voices via the
|
||||
# admin UI (see services/tts.py for the voice-discovery logic).
|
||||
# Voice catalog: https://huggingface.co/rhasspy/piper-voices
|
||||
RUN mkdir -p /opt/piper-voices && cd /opt/piper-voices && \
|
||||
for VOICE in en_US-amy-medium en_US-ryan-medium; do \
|
||||
DATASET="${VOICE#en_US-}"; DATASET="${DATASET%-medium}"; \
|
||||
BASE="https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/${DATASET}/medium"; \
|
||||
curl -fsSL -o "${VOICE}.onnx" "${BASE}/${VOICE}.onnx"; \
|
||||
curl -fsSL -o "${VOICE}.onnx.json" "${BASE}/${VOICE}.onnx.json"; \
|
||||
done
|
||||
|
||||
# Build the fable-mcp wheel so it can be served for download
|
||||
COPY fable-mcp/ fable-mcp/
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
"""reset voice_tts_voice and voice_tts_blend settings for piper migration
|
||||
|
||||
Revision ID: 0047
|
||||
Revises: 0046
|
||||
Create Date: 2026-05-22
|
||||
|
||||
The TTS backend swapped from kokoro to piper. Kokoro voice IDs
|
||||
(`af_heart`, `am_adam`, etc.) don't map to piper voice files
|
||||
(`en_US-amy-medium`, etc.) and there's no sensible auto-translation.
|
||||
Clear the stored voice selection so every user falls back to the piper
|
||||
default the next time they synthesize.
|
||||
|
||||
`voice_tts_blend` goes away entirely — piper has no voice-blending
|
||||
equivalent. The TTS service accepts the field for backward compat but
|
||||
ignores it; clearing the rows makes the DB match reality.
|
||||
|
||||
Both settings re-default cleanly on read, so this migration just deletes
|
||||
the rows without backfilling anything.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision = "0047"
|
||||
down_revision = "0046"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"DELETE FROM settings WHERE key IN ('voice_tts_voice', 'voice_tts_blend')"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# No-op. The deleted settings just re-default on read; restoring the
|
||||
# kokoro IDs wouldn't help anyway since kokoro is gone.
|
||||
pass
|
||||
+10
-13
@@ -623,11 +623,9 @@ export interface VoiceStatusResult {
|
||||
export interface VoiceEntry {
|
||||
id: string
|
||||
label: string
|
||||
}
|
||||
|
||||
export interface VoiceBlendEntry {
|
||||
voice: string
|
||||
weight: number
|
||||
language?: string
|
||||
quality?: string
|
||||
sample_rate?: number
|
||||
}
|
||||
|
||||
export const getVoiceStatus = () => apiGet<VoiceStatusResult>('/api/voice/status')
|
||||
@@ -651,16 +649,15 @@ export async function synthesiseSpeech(
|
||||
text: string,
|
||||
voice?: string,
|
||||
speed?: number,
|
||||
voiceBlend?: VoiceBlendEntry[]
|
||||
): Promise<Blob> {
|
||||
// Only send voice/speed/blend when explicitly provided — omitting them lets
|
||||
// the server auto-load the user's saved voice settings (voice, speed, blend).
|
||||
// Only send voice/speed when explicitly provided — omitting them lets the
|
||||
// server auto-load the user's saved voice settings.
|
||||
// Voice blending was removed with the kokoro → piper migration (piper has
|
||||
// no blend equivalent); callers that previously passed `voiceBlend` should
|
||||
// pass the first voice's id as `voice` instead.
|
||||
const body: Record<string, unknown> = { text }
|
||||
if (voiceBlend && voiceBlend.length >= 2) {
|
||||
body.voice_blend = voiceBlend
|
||||
if (speed !== undefined) body.speed = speed
|
||||
} else if (voice !== undefined || speed !== undefined) {
|
||||
body.voice = voice ?? 'af_heart'
|
||||
if (voice !== undefined || speed !== undefined) {
|
||||
body.voice = voice ?? 'en_US-amy-medium'
|
||||
body.speed = speed ?? 1.0
|
||||
}
|
||||
const res = await fetch('/api/voice/synthesise', {
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
<script setup lang="ts">
|
||||
import { X } from "lucide-vue-next";
|
||||
import { ref, computed, watch, onMounted } from "vue";
|
||||
import { useSettingsStore } from "@/stores/settings";
|
||||
import { useAuthStore } from "@/stores/auth";
|
||||
import { useToastStore } from "@/stores/toast";
|
||||
import { apiGet, apiPost, apiPut, apiDelete, listGroups, createGroup, deleteGroup, listGroupMembers, addGroupMember, removeGroupMember, searchUsers, getFableMcpInfo, listApiKeys, createApiKey as apiCreateApiKey, revokeApiKey as apiRevokeApiKey, getVoiceStatus, getVoiceList, synthesiseSpeech, getProfile, updateProfile, consolidateProfile, clearProfileObservations, listProfileObservations, getJournalConfig, saveJournalConfig, geocodeAddress, type ApiKeyEntry, type GroupEntry, type GroupMember, type UserSearchResult, type VoiceStatusResult, type VoiceEntry, type VoiceBlendEntry, type UserProfile, type JournalConfig, type ProfileObservationEntry } from "@/api/client";
|
||||
import { apiGet, apiPost, apiPut, apiDelete, listGroups, createGroup, deleteGroup, listGroupMembers, addGroupMember, removeGroupMember, searchUsers, getFableMcpInfo, listApiKeys, createApiKey as apiCreateApiKey, revokeApiKey as apiRevokeApiKey, getVoiceStatus, getVoiceList, synthesiseSpeech, getProfile, updateProfile, consolidateProfile, clearProfileObservations, listProfileObservations, getJournalConfig, saveJournalConfig, geocodeAddress, type ApiKeyEntry, type GroupEntry, type GroupMember, type UserSearchResult, type VoiceStatusResult, type VoiceEntry, type UserProfile, type JournalConfig, type ProfileObservationEntry } from "@/api/client";
|
||||
import { usePushStore } from "@/stores/push";
|
||||
import type { User } from "@/types/auth";
|
||||
import PaginationBar from "@/components/PaginationBar.vue";
|
||||
@@ -457,58 +456,19 @@ const searchResults = ref<{ url: string; title: string; snippet: string }[]>([])
|
||||
const searchLoading = ref(false);
|
||||
const searchError = ref("");
|
||||
|
||||
// Voice settings
|
||||
// Voice settings. Default voice id matches the bundled piper voice
|
||||
// (en_US-amy-medium); legacy kokoro IDs in user settings were cleared
|
||||
// by alembic migration 0047, so the first read after upgrade returns "".
|
||||
const voiceStatus = ref<VoiceStatusResult | null>(null);
|
||||
const voiceStatusLoading = ref(false);
|
||||
const availableVoices = ref<VoiceEntry[]>([]);
|
||||
const voiceTtsVoice = ref("af_heart");
|
||||
const voiceTtsVoice = ref("en_US-amy-medium");
|
||||
const voiceTtsSpeed = ref(1.0);
|
||||
const voiceSpeechStyle = ref("conversational");
|
||||
const savingVoice = ref(false);
|
||||
const voiceSaved = ref(false);
|
||||
const voiceTabLoaded = ref(false);
|
||||
|
||||
// Voice blend
|
||||
const blendEnabled = ref(false);
|
||||
const voiceBlend = ref<VoiceBlendEntry[]>([
|
||||
{ voice: "af_heart", weight: 1.0 },
|
||||
{ voice: "af_bella", weight: 1.0 },
|
||||
]);
|
||||
const blendPreviewText = ref("Hello! I'm your assistant. How can I help you today?");
|
||||
const blendPreviewing = ref(false);
|
||||
|
||||
function addBlendSlot() {
|
||||
voiceBlend.value.push({ voice: "af_heart", weight: 1.0 });
|
||||
}
|
||||
|
||||
function removeBlendSlot(idx: number) {
|
||||
if (voiceBlend.value.length > 2) voiceBlend.value.splice(idx, 1);
|
||||
}
|
||||
|
||||
async function previewBlend() {
|
||||
if (blendPreviewing.value || !blendPreviewText.value.trim()) return;
|
||||
blendPreviewing.value = true;
|
||||
// Create AudioContext synchronously in user-gesture context to bypass autoplay policy
|
||||
const ctx = new AudioContext();
|
||||
try {
|
||||
const blob = await synthesiseSpeech(
|
||||
blendPreviewText.value,
|
||||
undefined,
|
||||
voiceTtsSpeed.value,
|
||||
voiceBlend.value,
|
||||
);
|
||||
const buf = await ctx.decodeAudioData(await blob.arrayBuffer());
|
||||
const src = ctx.createBufferSource();
|
||||
src.buffer = buf;
|
||||
src.connect(ctx.destination);
|
||||
src.start(0);
|
||||
} catch {
|
||||
toastStore.show("Preview failed — TTS may not be ready", "error");
|
||||
} finally {
|
||||
blendPreviewing.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function loadVoiceTab() {
|
||||
if (voiceTabLoaded.value) return;
|
||||
voiceTabLoaded.value = true;
|
||||
@@ -558,7 +518,6 @@ async function saveVoiceSettings() {
|
||||
voice_tts_voice: voiceTtsVoice.value,
|
||||
voice_tts_speed: String(voiceTtsSpeed.value),
|
||||
voice_speech_style: voiceSpeechStyle.value,
|
||||
voice_tts_blend: blendEnabled.value ? JSON.stringify(voiceBlend.value) : "",
|
||||
});
|
||||
voiceSaved.value = true;
|
||||
setTimeout(() => { voiceSaved.value = false; }, 2000);
|
||||
@@ -805,19 +764,12 @@ onMounted(async () => {
|
||||
// Load journal config (locations, temp unit, prep schedule)
|
||||
await loadJournalConfig();
|
||||
|
||||
// Load voice settings
|
||||
// Load voice settings. Voice blending was removed with the kokoro→piper
|
||||
// migration (piper has no blend equivalent); legacy voice_tts_blend rows
|
||||
// were dropped in migration 0047.
|
||||
if (allSettings.voice_tts_voice) voiceTtsVoice.value = allSettings.voice_tts_voice;
|
||||
if (allSettings.voice_tts_speed) voiceTtsSpeed.value = parseFloat(allSettings.voice_tts_speed);
|
||||
if (allSettings.voice_speech_style) voiceSpeechStyle.value = allSettings.voice_speech_style;
|
||||
if (allSettings.voice_tts_blend) {
|
||||
try {
|
||||
const parsed = JSON.parse(allSettings.voice_tts_blend);
|
||||
if (Array.isArray(parsed) && parsed.length >= 2) {
|
||||
voiceBlend.value = parsed;
|
||||
blendEnabled.value = true;
|
||||
}
|
||||
} catch { /* ignore malformed */ }
|
||||
}
|
||||
|
||||
// Load CalDAV settings
|
||||
try {
|
||||
@@ -2407,70 +2359,8 @@ function formatUserDate(iso: string): string {
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section v-if="voiceStatus?.enabled && voiceStatus?.tts" class="settings-section full-width">
|
||||
<h2>Voice Blend</h2>
|
||||
<p class="section-desc">
|
||||
Mix two or more voice styles by weight. The resulting blended voice is used instead of
|
||||
the single voice above when enabled.
|
||||
</p>
|
||||
|
||||
<div class="field-group">
|
||||
<label class="toggle-label">
|
||||
<input type="checkbox" v-model="blendEnabled" />
|
||||
<span>Enable voice blending</span>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<template v-if="blendEnabled">
|
||||
<div class="blend-slots">
|
||||
<div v-for="(entry, idx) in voiceBlend" :key="idx" class="blend-slot">
|
||||
<select class="input blend-voice-select" v-model="entry.voice">
|
||||
<option v-for="v in availableVoices" :key="v.id" :value="v.id">{{ v.label }}</option>
|
||||
</select>
|
||||
<div class="blend-weight-row">
|
||||
<input
|
||||
type="range"
|
||||
min="0.1"
|
||||
max="2.0"
|
||||
step="0.05"
|
||||
v-model.number="entry.weight"
|
||||
class="range-input blend-weight-slider"
|
||||
/>
|
||||
<span class="blend-weight-label">{{ entry.weight.toFixed(2) }}</span>
|
||||
<button
|
||||
v-if="voiceBlend.length > 2"
|
||||
class="btn-remove-slot"
|
||||
@click="removeBlendSlot(idx)"
|
||||
title="Remove this voice"
|
||||
><X :size="16" /></button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="blend-actions">
|
||||
<button class="btn-secondary" @click="addBlendSlot" :disabled="voiceBlend.length >= 5">
|
||||
+ Add voice
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="field-group" style="margin-top: 1rem;">
|
||||
<label class="field-label">Preview text</label>
|
||||
<input
|
||||
type="text"
|
||||
class="input"
|
||||
v-model="blendPreviewText"
|
||||
placeholder="Text to speak for preview…"
|
||||
maxlength="300"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div class="form-actions">
|
||||
<button class="btn-secondary" @click="previewBlend" :disabled="blendPreviewing">
|
||||
{{ blendPreviewing ? 'Generating…' : '▶ Preview blend' }}
|
||||
</button>
|
||||
</div>
|
||||
</template>
|
||||
</section>
|
||||
<!-- Voice Blend section removed with the kokoro → piper migration
|
||||
(2026-05-22). Piper has no voice-blending equivalent. -->
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
@@ -30,22 +30,25 @@ async def voice_status():
|
||||
"stt": stt_available(),
|
||||
"tts": tts_available(),
|
||||
"stt_model": config.get("voice_stt_model", "base.en"),
|
||||
"tts_backend": "kokoro",
|
||||
"tts_backend": "piper",
|
||||
})
|
||||
|
||||
|
||||
@voice_bp.route("/voices", methods=["GET"])
|
||||
@login_required
|
||||
async def list_voices():
|
||||
"""Return available Kokoro voice IDs and labels."""
|
||||
"""Return available piper voice IDs and metadata.
|
||||
|
||||
Scans /opt/piper-voices (bundled) + /data/voices (admin-downloaded)
|
||||
on every call so newly-downloaded voices show up without a restart.
|
||||
Does NOT require tts_available() — even if the active voice failed
|
||||
to load, the catalog is still useful for picking a different one.
|
||||
"""
|
||||
from fabledassistant.services.voice_config import is_voice_enabled
|
||||
if not await is_voice_enabled():
|
||||
return jsonify({"error": "Voice feature is disabled"}), 503
|
||||
|
||||
from fabledassistant.services.tts import list_voices, tts_available
|
||||
|
||||
if not tts_available():
|
||||
return jsonify({"error": "TTS not available"}), 503
|
||||
from fabledassistant.services.tts import list_voices
|
||||
|
||||
return jsonify({"voices": list_voices()})
|
||||
|
||||
@@ -127,21 +130,18 @@ async def synthesise_speech():
|
||||
)
|
||||
return jsonify({"error": "text too long (max 8000 characters)"}), 400
|
||||
|
||||
voice = str(data.get("voice", "af_heart"))
|
||||
# Piper voice file basename (e.g. "en_US-amy-medium"). Default is read
|
||||
# from user settings if not in the request body.
|
||||
voice = str(data.get("voice", "")) or "en_US-amy-medium"
|
||||
try:
|
||||
speed = float(data.get("speed", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
speed = 1.0
|
||||
|
||||
voice_blend = data.get("voice_blend")
|
||||
if not isinstance(voice_blend, list) or len(voice_blend) < 2:
|
||||
voice_blend = None
|
||||
|
||||
# When no explicit voice/blend/speed provided, load all voice settings from the user's profile
|
||||
if "voice" not in data and "voice_blend" not in data and "speed" not in data:
|
||||
# Pull saved settings only when caller didn't override.
|
||||
if "voice" not in data and "speed" not in data:
|
||||
from fabledassistant.services.settings import get_setting
|
||||
from fabledassistant.auth import get_current_user_id
|
||||
import json as _json
|
||||
try:
|
||||
uid = get_current_user_id()
|
||||
saved_voice = await get_setting(uid, "voice_tts_voice", "")
|
||||
@@ -153,23 +153,17 @@ async def synthesise_speech():
|
||||
speed = float(saved_speed)
|
||||
except ValueError:
|
||||
pass
|
||||
saved_blend = await get_setting(uid, "voice_tts_blend", "")
|
||||
if saved_blend:
|
||||
parsed = _json.loads(saved_blend)
|
||||
if isinstance(parsed, list) and len(parsed) >= 2:
|
||||
voice_blend = parsed
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
blend_desc = f"blend({len(voice_blend)} voices)" if voice_blend else voice
|
||||
logger.info("TTS synthesis start: %d chars, voice=%s, speed=%.2f", char_count, blend_desc, speed)
|
||||
logger.info("TTS synthesis start: %d chars, voice=%s, speed=%.2f", char_count, voice, speed)
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
wav_bytes = await synthesise(text, voice=voice, speed=speed, voice_blend=voice_blend)
|
||||
wav_bytes = await synthesise(text, voice=voice, speed=speed)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"TTS synthesis failed: %d chars, voice=%s. Preview: %r",
|
||||
char_count, blend_desc, text[:120],
|
||||
char_count, voice, text[:120],
|
||||
)
|
||||
return jsonify({"error": "Synthesis failed"}), 500
|
||||
|
||||
@@ -177,12 +171,12 @@ async def synthesise_speech():
|
||||
if not wav_bytes:
|
||||
logger.warning(
|
||||
"TTS synthesis returned empty audio: %d chars, voice=%s, %dms. Preview: %r",
|
||||
char_count, blend_desc, duration_ms, text[:120],
|
||||
char_count, voice, duration_ms, text[:120],
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"TTS synthesis complete: %d chars → %d bytes in %dms (voice=%s)",
|
||||
char_count, len(wav_bytes), duration_ms, blend_desc,
|
||||
char_count, len(wav_bytes), duration_ms, voice,
|
||||
)
|
||||
|
||||
from quart import Response
|
||||
|
||||
+220
-203
@@ -1,270 +1,287 @@
|
||||
"""Text-to-speech service using Kokoro TTS (in-process)."""
|
||||
"""Text-to-speech service using piper-tts (in-process, CPU).
|
||||
|
||||
Voice model files (.onnx + .onnx.json sidecar) live in two directories:
|
||||
|
||||
- /opt/piper-voices/ — bundled at image build time, read-only.
|
||||
- /data/voices/ — admin-downloaded at runtime, persisted across
|
||||
restarts via the same volume as other /data/*.
|
||||
|
||||
A single PiperVoice is kept loaded at a time (one voice per request would
|
||||
be too slow; keeping all warm would use a lot of RAM for a feature most
|
||||
users won't enable). The active voice is whichever one the user picked
|
||||
via Settings → Voice — selection changes trigger reload_tts_model().
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from kokoro import KPipeline
|
||||
from piper import PiperVoice
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_pipeline: "KPipeline | None" = None
|
||||
_pipeline_lock = asyncio.Lock()
|
||||
# Two directories scanned for voices. Order matters for tie-breaking when
|
||||
# both contain the same voice id — /data wins (admin-downloaded override).
|
||||
_BUNDLED_VOICES_DIR = Path("/opt/piper-voices")
|
||||
_USER_VOICES_DIR = Path("/data/voices")
|
||||
|
||||
_DEFAULT_VOICE_ID = "en_US-amy-medium"
|
||||
|
||||
_voice: "PiperVoice | None" = None
|
||||
_voice_id: str | None = None # id of the currently-loaded voice
|
||||
_voice_lock = asyncio.Lock()
|
||||
_load_error: str | None = None
|
||||
|
||||
# Repo identifier used for HuggingFace update checks
|
||||
KOKORO_REPO = "hexgrad/Kokoro-82M"
|
||||
|
||||
# Persisted across restarts so we can detect model updates and run offline otherwise
|
||||
_COMMIT_HASH_FILE = Path("/data/kokoro_commit_hash.txt")
|
||||
|
||||
# Static list of supported Kokoro voice IDs and display labels
|
||||
_VOICES: list[dict] = [
|
||||
{"id": "af_heart", "label": "Heart (American Female, warm)"},
|
||||
{"id": "af_bella", "label": "Bella (American Female, expressive)"},
|
||||
{"id": "af_nicole", "label": "Nicole (American Female, intimate)"},
|
||||
{"id": "af_sarah", "label": "Sarah (American Female, clear)"},
|
||||
{"id": "af_sky", "label": "Sky (American Female, bright)"},
|
||||
{"id": "am_adam", "label": "Adam (American Male, neutral)"},
|
||||
{"id": "am_michael", "label": "Michael (American Male, deep)"},
|
||||
{"id": "bf_emma", "label": "Emma (British Female)"},
|
||||
{"id": "bf_isabella", "label": "Isabella (British Female, formal)"},
|
||||
{"id": "bm_george", "label": "George (British Male)"},
|
||||
{"id": "bm_lewis", "label": "Lewis (British Male, casual)"},
|
||||
]
|
||||
def _voice_dirs() -> list[Path]:
|
||||
"""Directories scanned for voice files, in precedence order (later wins)."""
|
||||
return [_BUNDLED_VOICES_DIR, _USER_VOICES_DIR]
|
||||
|
||||
|
||||
def _read_stored_commit() -> str | None:
|
||||
def _discover_voice(voice_id: str) -> Path | None:
|
||||
"""Find the .onnx file for `voice_id` across the scanned directories.
|
||||
|
||||
/data wins over /opt if both contain the same id so an admin-downloaded
|
||||
voice can override a bundled one if they want.
|
||||
"""
|
||||
found: Path | None = None
|
||||
for d in _voice_dirs():
|
||||
candidate = d / f"{voice_id}.onnx"
|
||||
if candidate.is_file() and (d / f"{voice_id}.onnx.json").is_file():
|
||||
found = candidate
|
||||
return found
|
||||
|
||||
|
||||
def _scan_voices() -> list[Path]:
|
||||
"""Return all valid voice files (.onnx + .onnx.json) across all dirs.
|
||||
|
||||
Same-id duplicates are deduplicated; later directories override earlier
|
||||
so a /data voice shadows a bundled one with the same name.
|
||||
"""
|
||||
by_id: dict[str, Path] = {}
|
||||
for d in _voice_dirs():
|
||||
if not d.is_dir():
|
||||
continue
|
||||
for onnx in sorted(d.glob("*.onnx")):
|
||||
sidecar = onnx.with_suffix(".onnx.json")
|
||||
if not sidecar.is_file():
|
||||
continue
|
||||
by_id[onnx.stem] = onnx
|
||||
return list(by_id.values())
|
||||
|
||||
|
||||
def _voice_metadata(onnx_path: Path) -> dict:
|
||||
"""Read the .onnx.json sidecar and return a UI-shaped metadata dict.
|
||||
|
||||
The sidecar's full schema is large; we surface only what the picker
|
||||
needs: id (filename stem), label (derived), language, quality, sample_rate.
|
||||
"""
|
||||
sidecar = onnx_path.with_suffix(".onnx.json")
|
||||
info: dict = {}
|
||||
try:
|
||||
if _COMMIT_HASH_FILE.exists():
|
||||
return _COMMIT_HASH_FILE.read_text().strip() or None
|
||||
info = json.loads(sidecar.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
logger.warning("Could not parse piper sidecar %s", sidecar, exc_info=True)
|
||||
|
||||
voice_id = onnx_path.stem
|
||||
lang_info = info.get("language") or {}
|
||||
audio_info = info.get("audio") or {}
|
||||
# Filename convention: <lang>-<dataset>-<quality> e.g. en_US-amy-medium
|
||||
parts = voice_id.split("-")
|
||||
dataset = parts[1] if len(parts) >= 2 else voice_id
|
||||
quality = parts[2] if len(parts) >= 3 else "medium"
|
||||
lang_code = lang_info.get("code") or (parts[0] if parts else "")
|
||||
region_name = lang_info.get("name_native") or lang_info.get("region") or lang_code
|
||||
|
||||
def _write_stored_commit(sha: str) -> None:
|
||||
try:
|
||||
_COMMIT_HASH_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
_COMMIT_HASH_FILE.write_text(sha)
|
||||
except Exception:
|
||||
logger.warning("Failed to write Kokoro commit hash to %s", _COMMIT_HASH_FILE)
|
||||
|
||||
|
||||
def _set_hf_offline(offline: bool) -> None:
|
||||
"""Toggle HuggingFace offline mode in the current process environment."""
|
||||
if offline:
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
else:
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
return {
|
||||
"id": voice_id,
|
||||
"label": f"{dataset.capitalize()} ({region_name}, {quality})",
|
||||
"language": lang_code,
|
||||
"quality": quality,
|
||||
"sample_rate": audio_info.get("sample_rate", 22050),
|
||||
}
|
||||
|
||||
|
||||
async def load_tts_model() -> None:
|
||||
"""Load the Kokoro pipeline. Called once at startup when voice is enabled."""
|
||||
global _pipeline, _load_error
|
||||
"""Load the active piper voice. Called once at startup if voice is enabled.
|
||||
|
||||
The active voice is determined by the per-user `voice_tts_voice` setting
|
||||
when known; falls back to `_DEFAULT_VOICE_ID` if unset or invalid.
|
||||
Because the active voice is per-user but the loaded model is a single
|
||||
in-process resource, this loads the default at startup; per-user voice
|
||||
selection triggers a reload at request time.
|
||||
"""
|
||||
global _voice, _voice_id, _load_error
|
||||
from fabledassistant.services.voice_config import is_voice_enabled
|
||||
|
||||
if not await is_voice_enabled():
|
||||
return
|
||||
|
||||
async with _pipeline_lock:
|
||||
if _pipeline is not None:
|
||||
async with _voice_lock:
|
||||
if _voice is not None:
|
||||
return
|
||||
try:
|
||||
from kokoro import KPipeline
|
||||
target_id = _DEFAULT_VOICE_ID
|
||||
onnx_path = _discover_voice(target_id)
|
||||
if onnx_path is None:
|
||||
# No bundled default? Fall back to whatever is on disk.
|
||||
discovered = _scan_voices()
|
||||
if not discovered:
|
||||
_load_error = (
|
||||
"No piper voice models found in /opt/piper-voices "
|
||||
"or /data/voices"
|
||||
)
|
||||
logger.error(_load_error)
|
||||
return
|
||||
onnx_path = discovered[0]
|
||||
target_id = onnx_path.stem
|
||||
logger.warning(
|
||||
"Default voice %r not found; using %r instead",
|
||||
_DEFAULT_VOICE_ID, target_id,
|
||||
)
|
||||
|
||||
# If we have a stored commit hash the model files are already cached
|
||||
# locally — run offline to skip HuggingFace network validation entirely.
|
||||
already_cached = _read_stored_commit() is not None
|
||||
if already_cached:
|
||||
_set_hf_offline(True)
|
||||
logger.info("Kokoro model previously cached — loading in offline mode")
|
||||
|
||||
logger.info("Loading Kokoro TTS pipeline...")
|
||||
logger.info("Loading piper TTS voice %s from %s", target_id, onnx_path)
|
||||
loop = asyncio.get_running_loop()
|
||||
from piper import PiperVoice
|
||||
|
||||
def _load_and_warm():
|
||||
p = KPipeline(lang_code="a") # "a" = American English
|
||||
# Pre-load all voice tensors so synthesis never hits HuggingFace at request time
|
||||
for v in _VOICES:
|
||||
try:
|
||||
p.load_voice(v["id"])
|
||||
except Exception:
|
||||
pass
|
||||
return p
|
||||
|
||||
_pipeline = await loop.run_in_executor(None, _load_and_warm)
|
||||
|
||||
# Record the current commit hash on first successful load so future
|
||||
# restarts know the model is cached and can skip HF network checks.
|
||||
if not already_cached:
|
||||
asyncio.create_task(_record_initial_commit())
|
||||
|
||||
logger.info("Kokoro TTS pipeline loaded and voices pre-warmed")
|
||||
_voice = await loop.run_in_executor(
|
||||
None, lambda: PiperVoice.load(str(onnx_path))
|
||||
)
|
||||
_voice_id = target_id
|
||||
logger.info("Piper TTS voice %s loaded", target_id)
|
||||
except Exception:
|
||||
_load_error = "Failed to load Kokoro TTS pipeline"
|
||||
_load_error = "Failed to load piper TTS voice"
|
||||
logger.exception(_load_error)
|
||||
|
||||
|
||||
async def _record_initial_commit() -> None:
|
||||
"""Fetch and store the current Kokoro commit hash after first download."""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
async def reload_tts_model(voice_id: str | None = None) -> None:
|
||||
"""Unload the current voice and load `voice_id` (or re-load the default).
|
||||
|
||||
def _fetch():
|
||||
from huggingface_hub import model_info
|
||||
return model_info(KOKORO_REPO).sha
|
||||
|
||||
sha = await loop.run_in_executor(None, _fetch)
|
||||
if sha:
|
||||
_write_stored_commit(sha)
|
||||
# Now that files are cached, switch to offline mode for subsequent runs
|
||||
_set_hf_offline(True)
|
||||
logger.info("Kokoro commit hash stored (%s) — future restarts will use offline mode", sha[:8])
|
||||
except Exception:
|
||||
logger.warning("Could not record Kokoro commit hash", exc_info=True)
|
||||
|
||||
|
||||
async def check_for_kokoro_updates() -> None:
|
||||
"""Check HuggingFace for Kokoro model updates.
|
||||
|
||||
Intended to be called on a daily schedule. Temporarily lifts offline mode
|
||||
to query the HF API, compares the latest commit SHA against the locally
|
||||
stored one, and reloads the pipeline only if the model has changed.
|
||||
Safe to call at runtime — used when the user changes voice in Settings.
|
||||
"""
|
||||
from fabledassistant.services.voice_config import is_voice_enabled
|
||||
if not await is_voice_enabled():
|
||||
return
|
||||
|
||||
stored_sha = _read_stored_commit()
|
||||
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _fetch_latest_sha():
|
||||
# Temporarily lift offline mode to reach the HF API
|
||||
_set_hf_offline(False)
|
||||
try:
|
||||
from huggingface_hub import model_info
|
||||
return model_info(KOKORO_REPO).sha
|
||||
finally:
|
||||
# Restore offline mode regardless of outcome
|
||||
_set_hf_offline(True)
|
||||
|
||||
latest_sha = await loop.run_in_executor(None, _fetch_latest_sha)
|
||||
except Exception:
|
||||
logger.warning("Kokoro update check failed — will retry tomorrow", exc_info=True)
|
||||
# Ensure offline mode is restored if the executor raised before the finally
|
||||
_set_hf_offline(bool(stored_sha))
|
||||
return
|
||||
|
||||
if latest_sha == stored_sha:
|
||||
logger.debug("Kokoro model is up to date (sha: %s)", (latest_sha or "")[:8])
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Kokoro model update detected (%s → %s), reloading pipeline",
|
||||
(stored_sha or "none")[:8],
|
||||
(latest_sha or "")[:8],
|
||||
)
|
||||
# Go online for the reload so Kokoro can download the updated files
|
||||
_set_hf_offline(False)
|
||||
try:
|
||||
await reload_tts_model()
|
||||
if latest_sha:
|
||||
_write_stored_commit(latest_sha)
|
||||
logger.info("Kokoro model updated and reloaded successfully")
|
||||
except Exception:
|
||||
logger.exception("Kokoro pipeline reload after update failed")
|
||||
finally:
|
||||
_set_hf_offline(True)
|
||||
|
||||
|
||||
async def reload_tts_model() -> None:
|
||||
"""Unload and reload the TTS pipeline. Safe to call at runtime."""
|
||||
global _pipeline, _load_error
|
||||
async with _pipeline_lock:
|
||||
_pipeline = None
|
||||
global _voice, _voice_id, _load_error
|
||||
async with _voice_lock:
|
||||
_voice = None
|
||||
_voice_id = None
|
||||
_load_error = None
|
||||
await load_tts_model()
|
||||
if voice_id:
|
||||
await _switch_voice(voice_id)
|
||||
else:
|
||||
await load_tts_model()
|
||||
|
||||
|
||||
async def _switch_voice(voice_id: str) -> None:
|
||||
"""Internal: load a specific voice by id. No-op if already loaded."""
|
||||
global _voice, _voice_id, _load_error
|
||||
async with _voice_lock:
|
||||
if _voice is not None and _voice_id == voice_id:
|
||||
return
|
||||
onnx_path = _discover_voice(voice_id)
|
||||
if onnx_path is None:
|
||||
_load_error = f"Voice {voice_id!r} not found in /opt or /data"
|
||||
logger.warning(_load_error)
|
||||
return
|
||||
try:
|
||||
logger.info("Switching piper voice to %s", voice_id)
|
||||
loop = asyncio.get_running_loop()
|
||||
from piper import PiperVoice
|
||||
|
||||
_voice = await loop.run_in_executor(
|
||||
None, lambda: PiperVoice.load(str(onnx_path))
|
||||
)
|
||||
_voice_id = voice_id
|
||||
_load_error = None
|
||||
except Exception:
|
||||
_load_error = f"Failed to load piper voice {voice_id!r}"
|
||||
logger.exception(_load_error)
|
||||
|
||||
|
||||
def tts_available() -> bool:
|
||||
return _pipeline is not None
|
||||
return _voice is not None
|
||||
|
||||
|
||||
def list_voices() -> list[dict]:
|
||||
return _VOICES
|
||||
"""Return metadata for every voice on disk, in stable order."""
|
||||
return [_voice_metadata(p) for p in _scan_voices()]
|
||||
|
||||
|
||||
async def synthesise(
|
||||
text: str,
|
||||
voice: str = "af_heart",
|
||||
voice: str = _DEFAULT_VOICE_ID,
|
||||
speed: float = 1.0,
|
||||
voice_blend: list[dict] | None = None,
|
||||
voice_blend: list[dict] | None = None, # accepted for backcompat; ignored
|
||||
) -> bytes:
|
||||
"""Synthesise text to WAV bytes (24kHz, 16-bit mono). Runs in executor.
|
||||
"""Synthesise text → WAV bytes (PCM 16-bit mono). Runs in executor.
|
||||
|
||||
voice_blend is a list of {"voice": str, "weight": float} dicts.
|
||||
When provided with 2+ entries the voice style tensors are merged as a
|
||||
weighted average before synthesis. Weights are normalised automatically.
|
||||
`voice_blend` is accepted for backward compatibility with callers that
|
||||
still pass it, but piper has no voice-blending equivalent, so it is
|
||||
silently ignored. The first voice in the blend (if any) is used; if
|
||||
that doesn't exist on disk, the request uses the default voice.
|
||||
|
||||
`speed` is the kokoro-shaped multiplier (1.0=normal, 1.5=faster).
|
||||
Piper uses `length_scale` where smaller = faster, so the conversion
|
||||
is `length_scale = 1.0 / speed`.
|
||||
"""
|
||||
if _pipeline is None:
|
||||
raise RuntimeError("TTS pipeline not loaded")
|
||||
# Backcompat: if blend was passed, just use the first entry's voice.
|
||||
if voice_blend and isinstance(voice_blend, list) and voice_blend:
|
||||
first = voice_blend[0]
|
||||
if isinstance(first, dict) and first.get("voice"):
|
||||
voice = str(first["voice"])
|
||||
|
||||
speed = max(0.7, min(1.3, speed))
|
||||
# Lazy-load / switch if needed.
|
||||
if _voice is None or _voice_id != voice:
|
||||
await _switch_voice(voice)
|
||||
if _voice is None:
|
||||
raise RuntimeError(f"TTS voice {voice!r} not loaded: {_load_error}")
|
||||
|
||||
def _build_voice_param():
|
||||
"""Return either a blended style tensor or a single voice ID string."""
|
||||
if not voice_blend or len(voice_blend) < 2:
|
||||
return voice_blend[0]["voice"] if voice_blend else voice
|
||||
import numpy as np
|
||||
total_w = sum(max(0.0, e.get("weight", 1.0)) for e in voice_blend) or 1.0
|
||||
blended = None
|
||||
for entry in voice_blend:
|
||||
vid = entry.get("voice", "af_heart")
|
||||
w = max(0.0, entry.get("weight", 1.0)) / total_w
|
||||
vt = _pipeline.load_voice(vid) # type: ignore[union-attr]
|
||||
blended = vt * w if blended is None else blended + vt * w
|
||||
return blended
|
||||
speed = max(0.5, min(2.0, speed))
|
||||
length_scale = 1.0 / speed
|
||||
|
||||
def _run() -> bytes:
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
# piper.SynthesisConfig is optional; defaults are sensible.
|
||||
# We override length_scale only.
|
||||
from piper import SynthesisConfig
|
||||
|
||||
config = SynthesisConfig(length_scale=length_scale)
|
||||
|
||||
voice_param = _build_voice_param()
|
||||
t0 = time.monotonic()
|
||||
audio_chunks: list = []
|
||||
try:
|
||||
for _, _, audio in _pipeline(text, voice=voice_param, speed=speed): # type: ignore[misc]
|
||||
if audio is not None:
|
||||
audio_chunks.append(audio)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Kokoro pipeline error during synthesis: %d chars, preview=%r",
|
||||
len(text), text[:80],
|
||||
)
|
||||
raise
|
||||
|
||||
if not audio_chunks:
|
||||
logger.warning(
|
||||
"Kokoro produced no audio chunks: %d chars, preview=%r",
|
||||
len(text), text[:80],
|
||||
)
|
||||
return b""
|
||||
|
||||
combined = np.concatenate(audio_chunks)
|
||||
buf = io.BytesIO()
|
||||
sf.write(buf, combined, samplerate=24000, format="WAV", subtype="PCM_16")
|
||||
sample_rate = _voice.config.sample_rate # type: ignore[union-attr]
|
||||
total_samples = 0
|
||||
with wave.open(buf, "wb") as wav:
|
||||
wav.setnchannels(1)
|
||||
wav.setsampwidth(2) # 16-bit PCM
|
||||
wav.setframerate(sample_rate)
|
||||
try:
|
||||
for chunk in _voice.synthesize(text, config): # type: ignore[union-attr]
|
||||
pcm = chunk.audio_int16_bytes
|
||||
if not pcm:
|
||||
continue
|
||||
wav.writeframes(pcm)
|
||||
total_samples += len(pcm) // 2
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Piper synthesis error: %d chars, preview=%r",
|
||||
len(text), text[:80],
|
||||
)
|
||||
raise
|
||||
|
||||
elapsed = time.monotonic() - t0
|
||||
logger.info(
|
||||
"Kokoro synthesis: %d chars → %d samples (%.2fs, %.0f chars/s)",
|
||||
len(text), len(combined), elapsed, len(text) / elapsed if elapsed > 0 else 0,
|
||||
"Piper synthesis: %d chars → %d samples (%.2fs, %.0f chars/s, voice=%s)",
|
||||
len(text), total_samples, elapsed,
|
||||
len(text) / elapsed if elapsed > 0 else 0,
|
||||
_voice_id,
|
||||
)
|
||||
if not total_samples:
|
||||
return b""
|
||||
return buf.getvalue()
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
Reference in New Issue
Block a user