feat: Add Qwen3-TTS backend, real-time audio playback, and improved install
- Refactor tts_provider.py to support multiple backends (Chatterbox + Qwen3-TTS) with a common BaseTTSProvider interface and factory function - Add Qwen3TTSProvider with custom voice (9 speakers), voice cloning, and automatic resampling from 12kHz to 24kHz - Add graceful backend detection: get_available_backends() checks which TTS packages are installed without crashing - Add real-time audio playback queue in TTSController.synthesize_and_play() so TTS output plays through speakers as text arrives (not just at session end) - Add "Play" checkbox in TTS UI to enable/disable real-time audio playback - Add TTS engine selector (chatterbox/qwen3) with Qwen3-specific options (speaker selection, model size 0.6B/1.7B) - Update install.sh with --tts flag for one-stop TTS backend setup using pip (not uv), with fallback to source installation for Chatterbox on Python 3.12+ - Persist new TTS settings (backend, auto_play, qwen3_speaker, qwen3_model_size) - Update INSTALL_TTS.md and help dialogs with new backend information https://claude.ai/code/session_01965m1v3f1cK6gDV1JHioA8
+964 −283 · 10 files changed
-
INSTALL_TTS.md+77 −42 -
requirements.txt+7 −3 -
scripts/install.sh+188 −2 -
src/tts_controller.py+176 −78 -
src/tts_provider.py+354 −114 -
src/whispering_ui/bridge.py+25 −34 -
src/whispering_ui/components/help.py+9 −3 -
src/whispering_ui/components/sidebar.py+77 −0 -
src/whispering_ui/main.py+46 −7 -
src/whispering_ui/state.py+5 −0
@@ -1,71 +1,106 @@ # TTS Installation Guide -## Standard Installation (Recommended) +Whispering supports two TTS backends. Install one or both. -This is the correct installation process to avoid the pkuseg dependency conflict: +## Quick Install (Recommended) -```bash -# Step 1: Install all dependencies from requirements.txt -pip install -r requirements.txt +Use the install script with the `--tts` flag: -# Step 2: Install chatterbox-tts WITHOUT its dependencies -# (we already installed the correct versions in step 1) -pip install chatterbox-tts --no-deps +```bash +# Interactive - choose which backend(s) to install +./scripts/install.sh --tts -# Step 3: Test if it works -python -c "from chatterbox.tts import ChatterboxTTS; print('✓ ChatterboxTTS loaded successfully!')" +# Or install a specific backend directly +./scripts/install.sh --tts=chatterbox +./scripts/install.sh --tts=qwen3 +./scripts/install.sh --tts=all ``` -**Why this works:** -- `requirements.txt` has all the individual components chatterbox needs (torch, transformers, etc.) -- We install chatterbox-tts with `--no-deps` so it doesn't pull in conflicting versions (like pkuseg) -- You get full control over dependency versions +## Manual Installation -## Option 2: If Option 1 fails - Manual source installation +### Option A: Qwen3-TTS (Recommended for multilingual) + +Supports 10 languages (English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian) with 9 built-in voices and voice cloning. ```bash -# Clone the chatterbox source -git clone https://github.com/resemble-ai/chatterbox /tmp/chatterbox -cd /tmp/chatterbox +# Activate your virtual environment first +source .venv/bin/activate + +# Install Qwen3-TTS +pip install qwen-tts -# Edit setup.py or pyproject.toml to remove pkuseg from dependencies -# Then install -pip install -e . +# Optional: faster inference with flash attention +MAX_JOBS=4 pip install -U flash-attn --no-build-isolation + +# Verify +python -c "from qwen_tts import Qwen3TTSModel; print('Qwen3-TTS OK')" ``` -## Option 3: Copy source directly into project +### Option B: Chatterbox TTS (ResembleAI) + +English-focused with voice cloning support. Requires special handling on Python 3.12+. ```bash -# Clone chatterbox repo -git clone https://github.com/resemble-ai/chatterbox /tmp/chatterbox +# Activate your virtual environment first +source .venv/bin/activate -# Copy just the chatterbox module into your project -cp -r /tmp/chatterbox/chatterbox ./ +# Step 1: Install dependencies (already done if you ran install.sh) +pip install -r requirements.txt -# Now you can import it directly without pip install -``` +# Step 2: Install chatterbox WITHOUT its dependencies +# (avoids the pkuseg/distutils conflict on Python 3.12+) +pip install chatterbox-tts --no-deps -## Option 4: Alternative TTS (if chatterbox doesn't work) +# Verify +python -c "from chatterbox.tts import ChatterboxTTS; print('Chatterbox OK')" +``` -If chatterbox proves too difficult, we can switch to **Coqui TTS**: +**If `pip install chatterbox-tts --no-deps` fails:** ```bash -pip install TTS +# Install from source +git clone https://github.com/resemble-ai/chatterbox /tmp/chatterbox +pip install --no-deps -e /tmp/chatterbox ``` -Usage: -```python -from TTS.api import TTS -tts = TTS("tts_models/en/ljspeech/tacotron2-DDC") -tts.tts_to_file(text="Hello world", file_path="output.wav") -``` +## Selecting a Backend in the UI + +1. Open the **AI & TTS** panel (click the "AI & TTS" button) +2. Enable TTS with the checkbox +3. Under **Engine**, select either `chatterbox` or `qwen3` +4. For Qwen3: choose a **Speaker** and **Model Size** +5. Check **Play** to hear audio through speakers in real time +6. Check **Save** to also save audio files to `tts_output/` + +## Audio Playback + +When **Play** is enabled, TTS output is played through your speakers in real time as text is transcribed. This works with all TTS sources (Whisper, AI, Translation). + +The playback uses a queue so segments are played sequentially without blocking. + +## Troubleshooting -## Testing +### Python 3.12+ distutils error (Chatterbox) -After installation, test with: +If you see `ModuleNotFoundError: No module named 'distutils'`: +- Use `pip install chatterbox-tts --no-deps` (already handled by install.sh) +- Or switch to Qwen3-TTS which doesn't have this issue + +### No audio output + +- Check that `sounddevice` is installed: `pip install sounddevice` +- Check your system audio output device +- Try: `python -c "import sounddevice; print(sounddevice.query_devices())"` + +### GPU/CUDA errors + +- Run `python debug_cuda.py` for diagnostics +- Ensure PyTorch CUDA version matches your GPU driver +- For CPU mode, the app will auto-detect and fall back + +## Checking Backend Status ```bash -python tts_provider.py +cd src/ +python -c "from tts_provider import get_available_backends; print(get_available_backends())" ``` - -This should synthesize a test phrase and show success. @@ -9,21 +9,25 @@ sounddevice PyYAML>=6.0 python-dotenv>=1.0.0 -# TTS dependencies (individual components to avoid pkuseg conflict) -# After installing these, run: pip install chatterbox-tts --no-deps +# TTS shared dependencies (both Chatterbox and Qwen3-TTS need these) torch>=2.0.0 torchaudio>=2.0.0 soundfile>=0.12.0 pydub>=0.25.0 # For MP3 and other audio format support librosa>=0.11.0 -s3tokenizer safetensors>=0.5.3 transformers>=4.46.0 + +# Chatterbox-specific dependencies (install chatterbox itself separately) +# pip install chatterbox-tts --no-deps +s3tokenizer diffusers>=0.29.0 omegaconf>=2.3.0 resemble-perth>=1.0.1 conformer>=0.3.2 +# Qwen3-TTS: pip install qwen-tts (installs its own deps) + # Optional: for auto-type feature pyautogui pyperclip @@ -16,6 +16,32 @@ echo -e "${BLUE}Whispering Installation Script${NC}" echo -e "${BLUE}========================================${NC}" echo +# Parse arguments +INSTALL_TTS=false +TTS_BACKEND="" +for arg in "$@"; do + case $arg in + --tts) + INSTALL_TTS=true + ;; + --tts=*) + INSTALL_TTS=true + TTS_BACKEND="${arg#*=}" + ;; + --help|-h) + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Options:" + echo " --tts Install TTS backends (interactive selection)" + echo " --tts=chatterbox Install Chatterbox TTS backend" + echo " --tts=qwen3 Install Qwen3-TTS backend" + echo " --tts=all Install all TTS backends" + echo " -h, --help Show this help message" + exit 0 + ;; + esac +done + # Check Python version echo -e "${YELLOW}Checking Python version...${NC}" if ! command -v python3 &> /dev/null; then @@ -100,6 +126,7 @@ echo echo -e "${YELLOW}Creating project directories...${NC}" mkdir -p log_output mkdir -p tts_output +mkdir -p tts_voices echo -e "${GREEN}✓ Directories created${NC}" echo @@ -121,10 +148,166 @@ echo # Make scripts executable echo -e "${YELLOW}Making scripts executable...${NC}" chmod +x scripts/run.sh -chmod +x scripts/debug_env.sh +chmod +x scripts/debug_env.sh 2>/dev/null || true echo -e "${GREEN}✓ Scripts are executable${NC}" echo +# ======================================== +# TTS Installation +# ======================================== + +install_chatterbox() { + echo -e "${YELLOW}Installing Chatterbox TTS...${NC}" + echo -e "${BLUE}Note: Chatterbox requires specific dependency handling.${NC}" + + # Check Python version for distutils compatibility + if [ "$PYTHON_MAJOR" -eq 3 ] && [ "$PYTHON_MINOR" -ge 12 ]; then + echo -e "${YELLOW}Python $PYTHON_VERSION detected. Using --no-deps to avoid distutils issues.${NC}" + fi + + # Install chatterbox without dependencies (deps already in requirements.txt) + pip install chatterbox-tts --no-deps 2>/dev/null && { + echo -e "${GREEN}✓ Chatterbox TTS installed${NC}" + } || { + echo -e "${YELLOW}pip install failed. Trying source installation...${NC}" + # Fallback: clone and install from source + TEMP_DIR=$(mktemp -d) + git clone --depth 1 https://github.com/resemble-ai/chatterbox "$TEMP_DIR/chatterbox" 2>/dev/null && { + pip install --no-deps -e "$TEMP_DIR/chatterbox" 2>/dev/null && { + echo -e "${GREEN}✓ Chatterbox TTS installed from source${NC}" + } || { + # Last resort: copy the module directly + if [ -d "$TEMP_DIR/chatterbox/chatterbox" ]; then + cp -r "$TEMP_DIR/chatterbox/chatterbox" "$PROJECT_DIR/src/" + echo -e "${GREEN}✓ Chatterbox TTS module copied to src/${NC}" + else + echo -e "${RED}✗ Failed to install Chatterbox TTS${NC}" + echo -e "${BLUE} You can try manually: pip install chatterbox-tts --no-deps${NC}" + fi + } + } || { + echo -e "${RED}✗ Failed to clone Chatterbox repository${NC}" + } + rm -rf "$TEMP_DIR" + } + + # Verify installation + python -c "from chatterbox.tts import ChatterboxTTS; print(' ✓ Chatterbox import successful')" 2>/dev/null || { + echo -e "${YELLOW} Note: Chatterbox import test failed. It may still work at runtime.${NC}" + } + echo +} + +install_qwen3() { + echo -e "${YELLOW}Installing Qwen3-TTS...${NC}" + pip install qwen-tts 2>/dev/null && { + echo -e "${GREEN}✓ Qwen3-TTS installed${NC}" + } || { + echo -e "${YELLOW}pip install failed. Trying from source...${NC}" + TEMP_DIR=$(mktemp -d) + git clone --depth 1 https://github.com/QwenLM/Qwen3-TTS.git "$TEMP_DIR/qwen3-tts" 2>/dev/null && { + pip install -e "$TEMP_DIR/qwen3-tts" 2>/dev/null && { + echo -e "${GREEN}✓ Qwen3-TTS installed from source${NC}" + } || { + echo -e "${RED}✗ Failed to install Qwen3-TTS${NC}" + echo -e "${BLUE} You can try manually: pip install qwen-tts${NC}" + } + } || { + echo -e "${RED}✗ Failed to clone Qwen3-TTS repository${NC}" + } + rm -rf "$TEMP_DIR" + } + + # Optional: flash attention for faster inference + echo -e "${YELLOW}Do you want to install flash-attn for faster Qwen3-TTS inference? (y/N)${NC}" + echo -e "${BLUE}(Requires a compatible GPU and may take a while to compile)${NC}" + read -r flash_response + if [[ "$flash_response" =~ ^([yY][eE][sS]|[yY])$ ]]; then + echo -e "${YELLOW}Installing flash-attn...${NC}" + MAX_JOBS=4 pip install -U flash-attn --no-build-isolation 2>/dev/null && { + echo -e "${GREEN}✓ flash-attn installed${NC}" + } || { + echo -e "${YELLOW} flash-attn installation failed (optional - TTS will still work)${NC}" + } + fi + + # Verify installation + python -c "from qwen_tts import Qwen3TTSModel; print(' ✓ Qwen3-TTS import successful')" 2>/dev/null || { + echo -e "${YELLOW} Note: Qwen3-TTS import test failed. It may still work at runtime.${NC}" + } + echo +} + +# Handle TTS installation +if [ "$INSTALL_TTS" = true ]; then + echo -e "${BLUE}========================================${NC}" + echo -e "${BLUE}TTS Backend Installation${NC}" + echo -e "${BLUE}========================================${NC}" + echo + + if [ -n "$TTS_BACKEND" ]; then + case $TTS_BACKEND in + chatterbox) + install_chatterbox + ;; + qwen3) + install_qwen3 + ;; + all) + install_chatterbox + install_qwen3 + ;; + *) + echo -e "${RED}Unknown TTS backend: $TTS_BACKEND${NC}" + echo -e "${BLUE}Available: chatterbox, qwen3, all${NC}" + ;; + esac + else + # Interactive TTS selection + echo -e "${YELLOW}Which TTS backend(s) would you like to install?${NC}" + echo " 1) Chatterbox TTS (ResembleAI - voice cloning, English)" + echo " 2) Qwen3-TTS (Alibaba - multilingual, 10 languages, multiple voices)" + echo " 3) Both" + echo " 4) Skip TTS installation" + echo + read -r -p "Select (1-4): " tts_choice + + case $tts_choice in + 1) + install_chatterbox + ;; + 2) + install_qwen3 + ;; + 3) + install_chatterbox + install_qwen3 + ;; + 4|*) + echo -e "${BLUE}Skipping TTS installation.${NC}" + echo -e "${BLUE}You can install later with: ./scripts/install.sh --tts${NC}" + ;; + esac + fi +else + echo -e "${BLUE}TTS backends not installed by default.${NC}" + echo -e "${BLUE}To install TTS, run: ./scripts/install.sh --tts${NC}" + echo +fi + +# Verify TTS status +echo -e "${YELLOW}Checking TTS backend status...${NC}" +python -c " +from tts_provider import get_available_backends +backends = get_available_backends() +for name, avail in backends.items(): + status = '✓ installed' if avail else '✗ not installed' + print(f' {name}: {status}') +if not any(backends.values()): + print(' No TTS backends available. Install with: ./scripts/install.sh --tts') +" 2>/dev/null || echo -e "${YELLOW} Could not check TTS status (run from src/ directory)${NC}" +echo + # Installation complete echo -e "${GREEN}========================================${NC}" echo -e "${GREEN}Installation Complete!${NC}" @@ -137,7 +320,10 @@ echo -e " 3. See README.md for usage instructions" echo echo -e "${BLUE}Optional setup:${NC}" echo -e " - For AI features: Edit .env and set OPENROUTER_API_KEY" -echo -e " - For TTS: See INSTALL_TTS.md for setup instructions" +echo -e " - For TTS: ${GREEN}./scripts/install.sh --tts${NC}" +echo -e " Or install individually:" +echo -e " - Chatterbox: ${GREEN}pip install chatterbox-tts --no-deps${NC}" +echo -e " - Qwen3-TTS: ${GREEN}pip install qwen-tts${NC}" echo echo -e "${YELLOW}Note: Remember to activate the virtual environment before running:${NC}" echo -e "${GREEN} source .venv/bin/activate${NC}" @@ -2,10 +2,11 @@ """ TTS Controller module for orchestrating text-to-speech operations. -Handles text chunking, audio file generation, and voice management. +Handles text chunking, audio file generation, playback queue, and voice management. """ import os +import queue import re import threading from pathlib import Path @@ -14,7 +15,11 @@ from typing import Optional, Callable, Literal import numpy as np import soundfile as sf -from tts_provider import TTSProvider +from tts_provider import ( + BaseTTSProvider, + create_provider, + get_available_backends, +) class TTSController: @@ -24,8 +29,10 @@ class TTSController: Handles: - Intelligent text chunking (max 300 chars per synthesis) - Audio file generation (WAV/OGG) + - Audio playback queue (synthesize + play in real time) - Voice reference management - Async synthesis with callbacks + - Multiple backend support (Chatterbox, Qwen3-TTS) """ MAX_CHUNK_SIZE = 300 # Character limit per synthesis call @@ -34,30 +41,44 @@ class TTSController: self, device: Literal["cpu", "cuda", "auto"] = "auto", output_dir: str = "tts_output", - model_type: Literal["standard", "multilingual"] = "standard" + backend: str = "chatterbox", + model_type: Literal["standard", "multilingual"] = "standard", + qwen3_model_size: str = "1.7B", + qwen3_speaker: str = "Ryan", ): - """ - Initialize TTS controller. - - Args: - device: Compute device for TTS model - output_dir: Directory for saving audio files - model_type: TTS model variant - """ - self.provider = TTSProvider(device=device, model_type=model_type) + self.backend = backend self.output_dir = Path(output_dir) self.output_dir.mkdir(exist_ok=True) + self.provider: BaseTTSProvider = create_provider( + backend=backend, + device=device, + model_type=model_type, + qwen3_model_size=qwen3_model_size, + qwen3_speaker=qwen3_speaker, + ) + self.reference_voice_path: Optional[str] = None self.language = "en" self.exaggeration = 0.5 self.cfg = 0.5 + self.speaker = "" # For Qwen3-TTS speaker selection # Callbacks self.on_progress: Optional[Callable[[str], None]] = None self.on_complete: Optional[Callable[[str], None]] = None self.on_error: Optional[Callable[[str], None]] = None + # Playback queue for real-time TTS + self._playback_queue: queue.Queue[Optional[str]] = queue.Queue() + self._playback_thread: Optional[threading.Thread] = None + self._playback_stop = threading.Event() + self._is_playing = False + + @property + def is_playing(self) -> bool: + return self._is_playing + def set_reference_voice(self, audio_path: Optional[str]): """Set reference voice for voice cloning.""" if audio_path and not os.path.exists(audio_path): @@ -68,41 +89,52 @@ class TTSController: self, language: str = "en", exaggeration: float = 0.5, - cfg: float = 0.5 + cfg: float = 0.5, + speaker: str = "", ): """Configure synthesis parameters.""" self.language = language self.exaggeration = exaggeration self.cfg = cfg + self.speaker = speaker + + def switch_backend( + self, + backend: str, + device: Literal["cpu", "cuda", "auto"] = "auto", + model_type: str = "standard", + qwen3_model_size: str = "1.7B", + qwen3_speaker: str = "Ryan", + ): + """Switch to a different TTS backend. Unloads the current provider.""" + if self.provider: + self.provider.unload() + + self.backend = backend + self.provider = create_provider( + backend=backend, + device=device, + model_type=model_type, + qwen3_model_size=qwen3_model_size, + qwen3_speaker=qwen3_speaker, + ) def chunk_text(self, text: str) -> list[str]: """ - Intelligently chunk text into segments ≤ 300 characters. - + Intelligently chunk text into segments <= 300 characters. Splits at sentence boundaries when possible, otherwise at word boundaries. - - Args: - text: Input text to chunk - - Returns: - List of text chunks, each ≤ 300 characters """ if len(text) <= self.MAX_CHUNK_SIZE: return [text] chunks = [] - - # Split into sentences first - # Matches . ! ? followed by space or end of string sentence_pattern = r'([.!?]+[\s]+|[.!?]+$)' sentences = re.split(sentence_pattern, text) - # Recombine sentence text with punctuation cleaned_sentences = [] i = 0 while i < len(sentences): if i + 1 < len(sentences) and re.match(sentence_pattern, sentences[i + 1]): - # Combine sentence with its punctuation cleaned_sentences.append(sentences[i] + sentences[i + 1]) i += 2 elif sentences[i].strip(): @@ -111,25 +143,19 @@ class TTSController: else: i += 1 - # Now group sentences into chunks current_chunk = "" - for sentence in cleaned_sentences: sentence = sentence.strip() if not sentence: continue - # If single sentence exceeds limit, split at word boundaries if len(sentence) > self.MAX_CHUNK_SIZE: - # Save current chunk if any if current_chunk: chunks.append(current_chunk.strip()) current_chunk = "" - # Split long sentence into words words = sentence.split() word_chunk = "" - for word in words: if len(word_chunk) + len(word) + 1 <= self.MAX_CHUNK_SIZE: word_chunk += (" " if word_chunk else "") + word @@ -137,21 +163,16 @@ class TTSController: if word_chunk: chunks.append(word_chunk.strip()) word_chunk = word - if word_chunk: chunks.append(word_chunk.strip()) - # If adding sentence doesn't exceed limit, add to current chunk elif len(current_chunk) + len(sentence) + 1 <= self.MAX_CHUNK_SIZE: current_chunk += (" " if current_chunk else "") + sentence - - # Otherwise, save current chunk and start new one else: if current_chunk: chunks.append(current_chunk.strip()) current_chunk = sentence - # Don't forget last chunk if current_chunk: chunks.append(current_chunk.strip()) @@ -162,26 +183,19 @@ class TTSController: text: str, output_filename: str, file_format: Literal["wav", "ogg"] = "wav", - async_mode: bool = False + async_mode: bool = False, ) -> Optional[str]: """ Synthesize text to audio file. - Automatically chunks text if needed and concatenates audio. - Args: - text: Text to synthesize - output_filename: Output filename (without extension) - file_format: Output format (wav or ogg) - async_mode: Run in background thread - Returns: Path to generated file (None if async) """ if async_mode: thread = threading.Thread( target=self._synthesize_worker, - args=(text, output_filename, file_format) + args=(text, output_filename, file_format), ) thread.daemon = True thread.start() @@ -190,14 +204,10 @@ class TTSController: return self._synthesize_worker(text, output_filename, file_format) def _synthesize_worker( - self, - text: str, - output_filename: str, - file_format: str + self, text: str, output_filename: str, file_format: str ) -> Optional[str]: """Worker function for synthesis (can run async).""" try: - # Progress callback if self.on_progress: self.on_progress("Chunking text...") @@ -206,9 +216,7 @@ class TTSController: if self.on_progress: self.on_progress(f"Synthesizing {len(chunks)} chunk(s)...") - # Synthesize each chunk audio_segments = [] - for i, chunk in enumerate(chunks): if self.on_progress: self.on_progress(f"Chunk {i+1}/{len(chunks)}: {chunk[:50]}...") @@ -218,28 +226,26 @@ class TTSController: reference_audio_path=self.reference_voice_path, language=self.language, exaggeration=self.exaggeration, - cfg=self.cfg + cfg=self.cfg, + speaker=self.speaker, ) audio_segments.append(audio) - # Concatenate audio segments if len(audio_segments) > 1: full_audio = np.concatenate(audio_segments) else: full_audio = audio_segments[0] - # Save to file output_path = self.output_dir / f"{output_filename}.{file_format}" if self.on_progress: self.on_progress(f"Saving to {output_path}...") - # soundfile handles both WAV and OGG sf.write( str(output_path), full_audio, self.provider.sample_rate, - format=file_format.upper() + format=file_format.upper(), ) if self.on_complete: @@ -259,9 +265,6 @@ class TTSController: """ Synthesize text to numpy array (for playback without saving). - Args: - text: Text to synthesize - Returns: Tuple of (audio_array, sample_rate) or None on error """ @@ -275,11 +278,11 @@ class TTSController: reference_audio_path=self.reference_voice_path, language=self.language, exaggeration=self.exaggeration, - cfg=self.cfg + cfg=self.cfg, + speaker=self.speaker, ) audio_segments.append(audio) - # Concatenate if len(audio_segments) > 1: full_audio = np.concatenate(audio_segments) else: @@ -292,12 +295,117 @@ class TTSController: self.on_error(f"TTS synthesis failed: {e}") return None + # ------------------------------------------------------------------ + # Real-time playback queue + # ------------------------------------------------------------------ + + def synthesize_and_play(self, text: str, also_save: bool = False, + file_format: str = "wav"): + """ + Synthesize text and play it through the speakers. + Runs in background - queues the text for sequential playback. + + Args: + text: Text to synthesize and play + also_save: If True, also save to file + file_format: File format when saving + """ + if not text.strip(): + return + + self._playback_queue.put((text, also_save, file_format)) + + # Start playback thread if not running + if self._playback_thread is None or not self._playback_thread.is_alive(): + self._playback_stop.clear() + self._playback_thread = threading.Thread( + target=self._playback_loop, daemon=True + ) + self._playback_thread.start() + + def _playback_loop(self): + """Background thread that processes the playback queue.""" + import sounddevice as sd + + while not self._playback_stop.is_set(): + try: + item = self._playback_queue.get(timeout=1.0) + except queue.Empty: + # No more items - exit thread + break + + if item is None: + break + + text, also_save, file_format = item + try: + self._is_playing = True + + if self.on_progress: + self.on_progress("Synthesizing...") + + result = self.synthesize_to_array(text) + if result is None: + continue + + audio, sample_rate = result + + # Save to file if requested + if also_save: + import time + timestamp = time.strftime("%Y%m%d_%H%M%S") + filename = f"tts_{timestamp}" + output_path = self.output_dir / f"{filename}.{file_format}" + sf.write(str(output_path), audio, sample_rate, + format=file_format.upper()) + if self.on_complete: + self.on_complete(str(output_path)) + + # Play audio + if self.on_progress: + self.on_progress("Playing...") + + sd.play(audio, sample_rate) + sd.wait() + + if self.on_progress: + self.on_progress("") + + except Exception as e: + error_msg = f"TTS playback error: {e}" + if self.on_error: + self.on_error(error_msg) + else: + print(error_msg) + finally: + self._is_playing = False + + def stop_playback(self): + """Stop current playback and clear the queue.""" + self._playback_stop.set() + + # Drain the queue + while not self._playback_queue.empty(): + try: + self._playback_queue.get_nowait() + except queue.Empty: + break + + # Stop audio + try: + import sounddevice as sd + sd.stop() + except Exception: + pass + + self._is_playing = False + if __name__ == "__main__": # Test chunking controller = TTSController() - test_text = "This is a test. " * 50 # Long text + test_text = "This is a test. " * 50 print(f"Original text length: {len(test_text)}") chunks = controller.chunk_text(test_text) @@ -305,17 +413,7 @@ if __name__ == "__main__": for i, chunk in enumerate(chunks): print(f" {i+1}. [{len(chunk)} chars] {chunk[:60]}...") - # Test synthesis - short_text = "Hello world! This is a test of the text to speech system." - print(f"\n\nSynthesizing: {short_text}") - - try: - result = controller.synthesize_to_file( - short_text, - "test_output", - file_format="wav" - ) - if result: - print(f"Success! Saved to: {result}") - except Exception as e: - print(f"Error: {e}") + # Check backends + print("\n\nAvailable backends:") + for name, available in get_available_backends().items(): + print(f" {name}: {'YES' if available else 'no'}") @@ -1,11 +1,10 @@ #!/usr/bin/env python3 """ -TTS Provider module for ResembleAI ChatterboxTTS integration. -Handles text-to-speech synthesis with voice cloning support. +TTS Provider module with multi-backend support. +Supports ChatterboxTTS and Qwen3-TTS with graceful fallback. """ -import io import os import warnings from typing import Optional, Literal @@ -18,81 +17,124 @@ warnings.filterwarnings('ignore', category=FutureWarning) warnings.filterwarnings('ignore', category=UserWarning) -class TTSProvider: - """ - Wrapper for ChatterboxTTS with support for voice cloning and audio generation. +# --------------------------------------------------------------------------- +# Backend availability detection +# --------------------------------------------------------------------------- - Handles model initialization, voice reference management, and speech synthesis - with configurable parameters. - """ +def get_available_backends() -> dict[str, bool]: + """Check which TTS backends are installed and importable.""" + backends = {} - def __init__( - self, - device: Literal["cpu", "cuda", "auto"] = "auto", - model_type: Literal["standard", "multilingual"] = "standard" - ): - """ - Initialize TTS provider. + # Check Chatterbox + try: + import chatterbox # noqa: F401 + backends["chatterbox"] = True + except ImportError: + backends["chatterbox"] = False + + # Check Qwen3-TTS + try: + import qwen_tts # noqa: F401 + backends["qwen3"] = True + except ImportError: + backends["qwen3"] = False + + return backends + + +# --------------------------------------------------------------------------- +# Base provider +# --------------------------------------------------------------------------- - Args: - device: Compute device (cpu/cuda/auto) - model_type: Model variant (standard or multilingual) - """ +class BaseTTSProvider: + """Abstract base for TTS providers.""" + + sample_rate: int = 24000 + + def __init__(self, device: Literal["cpu", "cuda", "auto"] = "auto"): self.device = self._resolve_device(device) - self.model_type = model_type self.model = None - self.sample_rate = 24000 # ChatterboxTTS default sample rate - - # Model will be loaded lazily on first synthesis self._initialized = False - def _resolve_device(self, device: str) -> str: - """Resolve device string to actual device.""" + @staticmethod + def _resolve_device(device: str) -> str: if device == "auto": return "cuda" if torch.cuda.is_available() else "cpu" return device + def synthesize( + self, + text: str, + reference_audio_path: Optional[str] = None, + language: str = "en", + exaggeration: float = 0.5, + cfg: float = 0.5, + speaker: str = "", + ) -> tuple[np.ndarray, int]: + raise NotImplementedError + + def get_device(self) -> str: + return self.device + + def is_initialized(self) -> bool: + return self._initialized + + def unload(self): + if self.model is not None: + del self.model + self.model = None + self._initialized = False + if self.device == "cuda" and torch.cuda.is_available(): + torch.cuda.empty_cache() + + +# --------------------------------------------------------------------------- +# Chatterbox provider +# --------------------------------------------------------------------------- + +class ChatterboxProvider(BaseTTSProvider): + """ChatterboxTTS provider with voice cloning support.""" + + def __init__( + self, + device: Literal["cpu", "cuda", "auto"] = "auto", + model_type: Literal["standard", "multilingual"] = "standard", + ): + super().__init__(device) + self.model_type = model_type + self.sample_rate = 24000 + def _ensure_initialized(self): - """Lazy load the TTS model on first use.""" if self._initialized: return try: if self.model_type == "multilingual": from chatterbox.mtl_tts import ChatterboxMultilingualTTS - print(f"Loading multilingual TTS model on {self.device}...") + print(f"Loading multilingual ChatterboxTTS on {self.device}...") self.model = ChatterboxMultilingualTTS.from_pretrained(device=self.device) else: from chatterbox.tts import ChatterboxTTS - print(f"Loading TTS model on {self.device}...") + print(f"Loading ChatterboxTTS on {self.device}...") self.model = ChatterboxTTS.from_pretrained(device=self.device) self._initialized = True - print(f"✓ TTS model loaded successfully on {self.device}") + print(f"ChatterboxTTS loaded on {self.device}") except ImportError as e: - error_msg = ( - "ChatterboxTTS not installed.\n" + raise ImportError( + "ChatterboxTTS is not installed.\n" "Install with: pip install chatterbox-tts --no-deps\n" - "See INSTALL_TTS.md for details." - ) - raise ImportError(error_msg) from e + "Or run: ./scripts/install.sh --tts" + ) from e except RuntimeError as e: - # Check for common CUDA/cuDNN errors error_str = str(e) if "cuDNN" in error_str or "CUDA" in error_str: - error_msg = ( - f"Failed to initialize TTS model - CUDA/cuDNN error:\n{e}\n\n" - "This is likely a cuDNN version mismatch.\n" - "PyTorch comes with its own cuDNN - don't install nvidia-cudnn separately.\n" - "Check run.sh doesn't add conflicting cuDNN to LD_LIBRARY_PATH.\n" - "Run: python debug_cuda.py for detailed diagnostics." - ) - raise RuntimeError(error_msg) from e - else: - raise RuntimeError(f"Failed to initialize TTS model: {e}") from e - except Exception as e: - raise RuntimeError(f"Unexpected error initializing TTS: {e}") from e + raise RuntimeError( + f"CUDA/cuDNN error loading ChatterboxTTS:\n{e}\n\n" + "Check that PyTorch CUDA version matches your GPU driver." + ) from e + raise RuntimeError(f"Failed to load ChatterboxTTS: {e}") from e def synthesize( self, @@ -100,96 +142,294 @@ class TTSProvider: reference_audio_path: Optional[str] = None, language: str = "en", exaggeration: float = 0.5, - cfg: float = 0.5 + cfg: float = 0.5, + speaker: str = "", ) -> tuple[np.ndarray, int]: - """ - Synthesize speech from text. - - Args: - text: Text to synthesize (max 300 characters recommended) - reference_audio_path: Path to reference voice audio file (optional) - language: Language code (for multilingual model) - exaggeration: Expressiveness control (0.0-1.0, default 0.5) - cfg: Classifier-free guidance scale (0.0-1.0, default 0.5) - - Returns: - Tuple of (audio_array, sample_rate) - audio_array: numpy array of audio samples - sample_rate: sample rate in Hz - - Raises: - ValueError: If text exceeds safe length - """ - # Ensure model is loaded self._ensure_initialized() - # Validate text length (300 char limit) if len(text) > 300: raise ValueError( - f"Text too long ({len(text)} chars). Maximum 300 characters per call. " + f"Text too long ({len(text)} chars). Max 300 per call. " "Use TTSController for automatic chunking." ) - if not text.strip(): raise ValueError("Cannot synthesize empty text") + audio = self.model.generate( + text=text, + audio_prompt_path=reference_audio_path, + exaggeration=exaggeration, + cfg_weight=cfg, + temperature=0.8, + ) + + if isinstance(audio, torch.Tensor): + audio = audio.cpu().numpy() + if len(audio.shape) > 1: + audio = audio.flatten() + + return audio, self.sample_rate + + +# --------------------------------------------------------------------------- +# Qwen3-TTS provider +# --------------------------------------------------------------------------- + +# Language name mapping for Qwen3-TTS (requires full language names) +_QWEN3_LANG_MAP = { + "en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean", + "de": "German", "fr": "French", "ru": "Russian", "pt": "Portuguese", + "es": "Spanish", "it": "Italian", +} + +# Default speakers per language +_QWEN3_DEFAULT_SPEAKERS = { + "English": "Ryan", + "Chinese": "Vivian", + "Japanese": "Ono_Anna", + "Korean": "Sohee", +} + +QWEN3_SPEAKERS = [ + "Ryan", "Aiden", "Vivian", "Serena", "Uncle_Fu", + "Dylan", "Eric", "Ono_Anna", "Sohee", +] + +QWEN3_MODEL_SIZES = ["0.6B", "1.7B"] + + +class Qwen3TTSProvider(BaseTTSProvider): + """Qwen3-TTS provider with custom voice and voice cloning support.""" + + def __init__( + self, + device: Literal["cpu", "cuda", "auto"] = "auto", + model_size: str = "1.7B", + default_speaker: str = "Ryan", + ): + super().__init__(device) + self.sample_rate = 24000 # We'll resample from 12kHz to 24kHz + self._native_sample_rate = 12000 # Qwen3-TTS native rate + self.model_size = model_size + self.default_speaker = default_speaker + self._clone_model = None # Separate model for voice cloning + self._voice_clone_prompt = None + + def _get_model_name(self, clone: bool = False) -> str: + if clone: + return f"Qwen/Qwen3-TTS-12Hz-{self.model_size}-Base" + return f"Qwen/Qwen3-TTS-12Hz-{self.model_size}-CustomVoice" + + def _ensure_initialized(self): + if self._initialized: + return + try: - # Generate audio using ChatterboxTTS API - # API: generate(text, repetition_penalty, min_p, top_p, audio_prompt_path, exaggeration, cfg_weight, temperature) - audio = self.model.generate( - text=text, - audio_prompt_path=reference_audio_path, - exaggeration=exaggeration, - cfg_weight=cfg, - temperature=0.8, # Default from API - ) + from qwen_tts import Qwen3TTSModel - # Convert to numpy array if needed - if isinstance(audio, torch.Tensor): - audio = audio.cpu().numpy() + model_name = self._get_model_name(clone=False) + print(f"Loading Qwen3-TTS ({model_name}) on {self.device}...") - # Ensure correct shape (flatten if needed) - if len(audio.shape) > 1: - audio = audio.flatten() + # Determine dtype and attention implementation + dtype = torch.bfloat16 if self.device == "cuda" else torch.float32 + kwargs = {"device_map": f"{self.device}:0" if self.device == "cuda" else self.device} + kwargs["dtype"] = dtype - return audio, self.sample_rate + # Try flash attention if available + try: + import flash_attn # noqa: F401 + kwargs["attn_implementation"] = "flash_attention_2" + except ImportError: + pass + self.model = Qwen3TTSModel.from_pretrained(model_name, **kwargs) + self._initialized = True + print(f"Qwen3-TTS loaded on {self.device}") + + except ImportError as e: + raise ImportError( + "Qwen3-TTS is not installed.\n" + "Install with: pip install qwen-tts\n" + "Or run: ./scripts/install.sh --tts" + ) from e except Exception as e: - raise RuntimeError(f"TTS synthesis failed: {e}") from e + raise RuntimeError(f"Failed to load Qwen3-TTS: {e}") from e - def get_device(self) -> str: - """Get current compute device.""" - return self.device + def _ensure_clone_model(self): + """Load the Base model for voice cloning (separate from CustomVoice).""" + if self._clone_model is not None: + return - def is_initialized(self) -> bool: - """Check if model is loaded.""" - return self._initialized + try: + from qwen_tts import Qwen3TTSModel - def unload(self): - """Unload model to free memory.""" - if self.model is not None: - del self.model - self.model = None - self._initialized = False + model_name = self._get_model_name(clone=True) + print(f"Loading Qwen3-TTS clone model ({model_name})...") + + dtype = torch.bfloat16 if self.device == "cuda" else torch.float32 + kwargs = {"device_map": f"{self.device}:0" if self.device == "cuda" else self.device} + kwargs["dtype"] = dtype + + try: + import flash_attn # noqa: F401 + kwargs["attn_implementation"] = "flash_attention_2" + except ImportError: + pass - # Clear CUDA cache if applicable + self._clone_model = Qwen3TTSModel.from_pretrained(model_name, **kwargs) + print("Qwen3-TTS clone model loaded") + + except Exception as e: + raise RuntimeError(f"Failed to load Qwen3-TTS clone model: {e}") from e + + def _resample(self, audio: np.ndarray, orig_sr: int, target_sr: int) -> np.ndarray: + """Resample audio from orig_sr to target_sr.""" + if orig_sr == target_sr: + return audio + try: + import librosa + return librosa.resample(audio, orig_sr=orig_sr, target_sr=target_sr) + except ImportError: + # Simple linear interpolation fallback + ratio = target_sr / orig_sr + n_samples = int(len(audio) * ratio) + indices = np.linspace(0, len(audio) - 1, n_samples) + return np.interp(indices, np.arange(len(audio)), audio) + + def synthesize( + self, + text: str, + reference_audio_path: Optional[str] = None, + language: str = "en", + exaggeration: float = 0.5, + cfg: float = 0.5, + speaker: str = "", + ) -> tuple[np.ndarray, int]: + if not text.strip(): + raise ValueError("Cannot synthesize empty text") + + # Resolve language to Qwen3-TTS format + lang_name = _QWEN3_LANG_MAP.get(language, "English") + + if reference_audio_path and os.path.exists(reference_audio_path): + return self._synthesize_clone(text, reference_audio_path, lang_name) + else: + return self._synthesize_custom_voice(text, lang_name, speaker) + + def _synthesize_custom_voice( + self, text: str, language: str, speaker: str = "" + ) -> tuple[np.ndarray, int]: + self._ensure_initialized() + + if not speaker: + speaker = _QWEN3_DEFAULT_SPEAKERS.get(language, self.default_speaker) + + wavs, sr = self.model.generate_custom_voice( + text=text, + language=language, + speaker=speaker, + instruct="", + ) + + audio = wavs[0] + if isinstance(audio, torch.Tensor): + audio = audio.cpu().numpy() + if len(audio.shape) > 1: + audio = audio.flatten() + + # Resample to 24kHz for consistent output + audio = self._resample(audio, sr, self.sample_rate) + + return audio, self.sample_rate + + def _synthesize_clone( + self, text: str, ref_audio_path: str, language: str + ) -> tuple[np.ndarray, int]: + self._ensure_clone_model() + + wavs, sr = self._clone_model.generate_voice_clone( + text=text, + language=language, + ref_audio=ref_audio_path, + ref_text="", # Qwen3 can work without ref_text + ) + + audio = wavs[0] + if isinstance(audio, torch.Tensor): + audio = audio.cpu().numpy() + if len(audio.shape) > 1: + audio = audio.flatten() + + audio = self._resample(audio, sr, self.sample_rate) + + return audio, self.sample_rate + + def unload(self): + super().unload() + if self._clone_model is not None: + del self._clone_model + self._clone_model = None if self.device == "cuda" and torch.cuda.is_available(): torch.cuda.empty_cache() -if __name__ == "__main__": - # Simple test - print("Testing TTSProvider...") +# --------------------------------------------------------------------------- +# Factory +# --------------------------------------------------------------------------- - provider = TTSProvider(device="auto") - print(f"Device: {provider.get_device()}") +def create_provider( + backend: str = "chatterbox", + device: Literal["cpu", "cuda", "auto"] = "auto", + model_type: str = "standard", + qwen3_model_size: str = "1.7B", + qwen3_speaker: str = "Ryan", +) -> BaseTTSProvider: + """ + Create a TTS provider for the given backend. - test_text = "Hello! This is a test of the text to speech system." - print(f"Synthesizing: {test_text}") + Args: + backend: "chatterbox" or "qwen3" + device: Compute device + model_type: Chatterbox model type ("standard" or "multilingual") + qwen3_model_size: Qwen3 model size ("0.6B" or "1.7B") + qwen3_speaker: Default Qwen3 speaker name - try: - audio, sr = provider.synthesize(test_text) - print(f"Success! Generated {len(audio)} samples at {sr}Hz") - print(f"Duration: {len(audio)/sr:.2f} seconds") - except Exception as e: - print(f"Error: {e}") + Returns: + TTS provider instance + """ + if backend == "qwen3": + return Qwen3TTSProvider( + device=device, + model_size=qwen3_model_size, + default_speaker=qwen3_speaker, + ) + else: + return ChatterboxProvider(device=device, model_type=model_type) + + +if __name__ == "__main__": + print("Checking TTS backend availability...") + backends = get_available_backends() + for name, available in backends.items(): + status = "INSTALLED" if available else "not installed" + print(f" {name}: {status}") + + # Test with first available backend + for name, available in backends.items(): + if available: + print(f"\nTesting {name}...") + provider = create_provider(backend=name, device="auto") + print(f"Device: {provider.get_device()}") + + test_text = "Hello! This is a test of the text to speech system." + try: + audio, sr = provider.synthesize(test_text, language="en") + print(f"Generated {len(audio)} samples at {sr}Hz") + print(f"Duration: {len(audio)/sr:.2f} seconds") + except Exception as e: + print(f"Error: {e}") + break + else: + print("\nNo TTS backends installed. Install one with:") + print(" pip install chatterbox-tts --no-deps") + print(" pip install qwen-tts") @@ -480,19 +480,24 @@ class ProcessingBridge: def _finalize_tts_session(self): """Finalize TTS session if TTS is enabled.""" if not self.state.tts_enabled or not self.tts_session_text.strip(): + self.tts_session_text = "" return if not self.tts_controller: + self.tts_session_text = "" return - if self.state.tts_save_file and self.tts_session_id: + # If auto-play was on, segments were already played (and optionally saved) + # individually. Only save a combined session file if save is on + # and auto-play was off (meaning segments weren't saved yet). + if self.state.tts_save_file and not self.state.tts_auto_play and self.tts_session_id: try: filename = f"tts_session_{self.tts_session_id}" self.tts_controller.synthesize_to_file( text=self.tts_session_text.strip(), output_filename=filename, file_format=self.state.tts_format, - async_mode=True + async_mode=True, ) except Exception as e: print(f"TTS finalization error: {e}") @@ -562,71 +567,56 @@ class ProcessingBridge: if is_qa_mode and tts_source == 'ai' and state_attr == 'ai_text': # In Q&A mode, trigger TTS for complete AI response if self.state.tts_enabled and self.state.tts_source == 'ai': - self._trigger_qa_tts(new_segment) + self._trigger_tts_playback(new_segment) # Clear AI output after response is complete (keep whisper for context) # We'll do this after TTS is done speaking elif tts_source and self.state.tts_enabled and self.state.tts_source == tts_source: - # Normal TTS mode (non-Q&A) + # Normal TTS mode - synthesize and play in real time + if self.state.tts_auto_play: + self._trigger_tts_playback(new_segment) + # Also accumulate for session file save self.tts_session_text += new_segment + " " if autotype_mode and self.state.autotype_mode == autotype_mode and new_segment.strip(): self._autotype_text(new_segment) - def _trigger_qa_tts(self, text: str): - """Trigger TTS synthesis for Q&A response and set up playback.""" + def _trigger_tts_playback(self, text: str): + """Synthesize text and play it through speakers using the playback queue.""" if not self.tts_controller or not text.strip(): return try: - import time - # Generate unique filename for this Q&A response - timestamp = time.strftime("%Y%m%d_%H%M%S") - filename = f"qa_response_{timestamp}" + self.state.tts_status_message = "Queued for playback..." + self.state.tts_is_playing = True - # Synthesize to file - output_path = self.tts_controller.synthesize_to_file( + # Use the controller's playback queue for sequential, non-blocking playback + self.tts_controller.synthesize_and_play( text=text.strip(), - output_filename=filename, + also_save=self.state.tts_save_file, file_format=self.state.tts_format, - async_mode=False # Wait for completion ) - - if output_path: - # Store the audio file path for playback - self.state.tts_audio_file = output_path - self.state.tts_status_message = "Ready to play" - - # Auto-play the audio - self._play_tts_audio(output_path) except Exception as e: - print(f"Q&A TTS error: {e}") + print(f"TTS playback error: {e}") self.state.tts_status_message = f"TTS error: {str(e)[:30]}" + self.state.tts_is_playing = False def _play_tts_audio(self, audio_path: str): - """Play TTS audio file in background thread and clear AI output when done.""" + """Play a TTS audio file in background thread.""" def play_audio(): try: import sounddevice as sd import soundfile as sf - # Read audio file data, samplerate = sf.read(audio_path) - # Update state self.state.tts_is_playing = True self.state.tts_status_message = "Playing..." - # Play audio (blocking in this thread) sd.play(data, samplerate) sd.wait() - # Clear AI output after playback in Q&A mode - self.state.ai_text = "" - self._ai_committed = "" - - # Update state self.state.tts_is_playing = False self.state.tts_status_message = "Playback complete" except Exception as e: @@ -634,7 +624,6 @@ class ProcessingBridge: self.state.tts_status_message = f"Playback error: {str(e)[:30]}" self.state.tts_is_playing = False - # Start playback in background thread threading.Thread(target=play_audio, daemon=True).start() def manual_ai_trigger(self): @@ -652,8 +641,10 @@ class ProcessingBridge: self._play_tts_audio(self.state.tts_audio_file) def stop_qa_audio(self): - """Stop Q&A TTS audio playback.""" + """Stop all TTS audio playback.""" try: + if self.tts_controller: + self.tts_controller.stop_playback() import sounddevice as sd sd.stop() self.state.tts_is_playing = False @@ -59,13 +59,19 @@ See AI_SETUP.md for details.""", "tts": """**Enable TTS:** Convert text to speech -**Voice:** Browse=upload ref audio for cloning | Clear=default +**Engine:** Chatterbox (English, voice cloning) | Qwen3-TTS (multilingual, 9 voices) -**Save File:** Auto-save to tts_output/ with timestamp +**Qwen3 Options:** Speaker (Ryan, Aiden, Vivian, etc.) | Size (0.6B lighter, 1.7B better) + +**Voice:** Browse=upload ref audio for cloning | Clear=default voice + +**Play:** Hear TTS output through speakers in real time + +**Save:** Auto-save audio files to tts_output/ **Format:** WAV (lossless) | OGG (compressed) -**Setup:** See INSTALL_TTS.md""", +**Setup:** See INSTALL_TTS.md | Run: ./scripts/install.sh --tts""", "file_transcription": """**File Transcription:** Transcribe audio files (batch mode) @@ -555,6 +555,75 @@ def create_sidebar(state: AppState, bridge: ProcessingBridge, output_container=N if state.tts_available: with tts_section: + # Backend selection + with ui.row().classes('items-center w-full gap-1'): + ui.label('Engine:').classes('text-xs w-10') + # Build backend options from what's available + backend_options = [] + if state.tts_backends_available.get("chatterbox"): + backend_options.append("chatterbox") + if state.tts_backends_available.get("qwen3"): + backend_options.append("qwen3") + # Always show both options so user knows what's possible + if "chatterbox" not in backend_options: + backend_options.append("chatterbox") + if "qwen3" not in backend_options: + backend_options.append("qwen3") + + tts_backend_select = register_tts(ui.select( + options=backend_options, + value=state.tts_backend + ).classes('flex-grow').props('dense')) + + def on_backend_change(e): + state.tts_backend = e.value + if bridge.tts_controller: + bridge.tts_controller.switch_backend( + backend=e.value, + qwen3_model_size=state.tts_qwen3_model_size, + qwen3_speaker=state.tts_qwen3_speaker, + ) + # Show/hide Qwen3 options + qwen3_row.set_visibility(e.value == "qwen3") + + tts_backend_select.on_value_change(on_backend_change) + + # Qwen3-TTS specific options (speaker + model size) + qwen3_row = ui.row().classes('items-center w-full gap-1') + qwen3_row.set_visibility(state.tts_backend == "qwen3") + + with qwen3_row: + ui.label('Speaker:').classes('text-xs') + from tts_provider import QWEN3_SPEAKERS, QWEN3_MODEL_SIZES + qwen3_speaker_select = register_tts(ui.select( + options=QWEN3_SPEAKERS, + value=state.tts_qwen3_speaker + ).classes('flex-grow').props('dense')) + + def on_speaker_change(e): + state.tts_qwen3_speaker = e.value + if bridge.tts_controller: + bridge.tts_controller.set_parameters(speaker=e.value) + + qwen3_speaker_select.on_value_change(on_speaker_change) + + ui.label('Size:').classes('text-xs') + qwen3_size_select = register_tts(ui.select( + options=QWEN3_MODEL_SIZES, + value=state.tts_qwen3_model_size + ).classes('w-14').props('dense')) + + def on_size_change(e): + state.tts_qwen3_model_size = e.value + if bridge.tts_controller and state.tts_backend == "qwen3": + bridge.tts_controller.switch_backend( + backend="qwen3", + qwen3_model_size=e.value, + qwen3_speaker=state.tts_qwen3_speaker, + ) + + qwen3_size_select.on_value_change(on_size_change) + # Source selection - compact with ui.row().classes('items-center w-full gap-1'): ui.label('Src:').classes('text-xs w-10') @@ -585,6 +654,11 @@ def create_sidebar(state: AppState, bridge: ProcessingBridge, output_container=N # Output options - compact with ui.row().classes('items-center w-full gap-1'): ui.label('Out:').classes('text-xs w-10') + + tts_play_cb = register_tts(ui.checkbox('Play', value=state.tts_auto_play).props('dense')) + tts_play_cb.on_value_change(lambda e: setattr(state, 'tts_auto_play', e.value)) + tts_play_cb.tooltip('Play audio through speakers in real time') + tts_save_cb = register_tts(ui.checkbox('Save', value=state.tts_save_file).props('dense')) tts_save_cb.on_value_change(lambda e: setattr(state, 'tts_save_file', e.value)) @@ -599,6 +673,9 @@ def create_sidebar(state: AppState, bridge: ProcessingBridge, output_container=N tts_status_label.text = state.tts_status_message[:50] else: tts_status_label.text = '' + # Sync playback state from controller + if bridge.tts_controller: + state.tts_is_playing = bridge.tts_controller.is_playing ui.timer(0.2, update_tts_status) @@ -87,17 +87,43 @@ def main(): # Check TTS availability try: + from tts_provider import get_available_backends from tts_controller import TTSController - state.tts_available = True - # Load TTS settings - state.tts_enabled = settings.get("tts_enabled", False) - state.tts_source = settings.get("tts_source", "whisper") - state.tts_save_file = settings.get("tts_save_file", False) - state.tts_format = settings.get("tts_format", "wav") + backends = get_available_backends() + state.tts_backends_available = backends + state.tts_available = any(backends.values()) + + if state.tts_available: + # Load TTS settings + state.tts_enabled = settings.get("tts_enabled", False) + state.tts_backend = settings.get("tts_backend", "chatterbox") + state.tts_source = settings.get("tts_source", "whisper") + state.tts_auto_play = settings.get("tts_auto_play", True) + state.tts_save_file = settings.get("tts_save_file", False) + state.tts_format = settings.get("tts_format", "wav") + state.tts_qwen3_speaker = settings.get("tts_qwen3_speaker", "Ryan") + state.tts_qwen3_model_size = settings.get("tts_qwen3_model_size", "1.7B") + + # If saved backend is not installed, fall back to first available + if not backends.get(state.tts_backend): + for name, avail in backends.items(): + if avail: + state.tts_backend = name + break + + # Print backend status + for name, avail in backends.items(): + status = "available" if avail else "not installed" + print(f" TTS backend {name}: {status}") + else: + print("No TTS backends installed. TTS features disabled.") + print(" Install Chatterbox: pip install chatterbox-tts --no-deps") + print(" Install Qwen3-TTS: pip install qwen-tts") except Exception as e: print(f"TTS features not available: {e}") state.tts_available = False + state.tts_backends_available = {} # Create processing bridge bridge = ProcessingBridge(state) @@ -106,7 +132,16 @@ def main(): if state.tts_available: try: from tts_controller import TTSController - bridge.tts_controller = TTSController(device="auto", output_dir="tts_output") + bridge.tts_controller = TTSController( + device="auto", + output_dir="tts_output", + backend=state.tts_backend, + qwen3_model_size=state.tts_qwen3_model_size, + qwen3_speaker=state.tts_qwen3_speaker, + ) + # Wire up status callbacks + bridge.tts_controller.on_progress = lambda msg: setattr(state, 'tts_status_message', msg) + bridge.tts_controller.on_error = lambda msg: setattr(state, 'tts_status_message', msg) except Exception as e: print(f"Failed to initialize TTS controller: {e}") state.tts_available = False @@ -356,9 +391,13 @@ def main(): if state.tts_available: settings.set("tts_enabled", state.tts_enabled) + settings.set("tts_backend", state.tts_backend) settings.set("tts_source", state.tts_source) + settings.set("tts_auto_play", state.tts_auto_play) settings.set("tts_save_file", state.tts_save_file) settings.set("tts_format", state.tts_format) + settings.set("tts_qwen3_speaker", state.tts_qwen3_speaker) + settings.set("tts_qwen3_model_size", state.tts_qwen3_model_size) settings.save() @@ -42,7 +42,9 @@ class AppState: # === TTS Settings === tts_enabled: bool = False + tts_backend: str = "chatterbox" # "chatterbox" or "qwen3" tts_source: str = "whisper" # "whisper", "ai", or "translation" + tts_auto_play: bool = True # Auto-play TTS output through speakers tts_save_file: bool = False tts_format: str = "wav" tts_voice_reference: Optional[str] = None @@ -50,6 +52,9 @@ class AppState: tts_status_message: str = "" # TTS status message tts_audio_file: Optional[str] = None # Current TTS audio file for playback tts_is_playing: bool = False # TTS audio playback state + tts_qwen3_speaker: str = "Ryan" # Qwen3-TTS speaker name + tts_qwen3_model_size: str = "1.7B" # Qwen3 model size ("0.6B" or "1.7B") + tts_backends_available: dict = field(default_factory=dict) # {name: bool} # === Autotype Settings === autotype_mode: str = "Off" # "Off", "Whisper", "Translation", "AI"