What is Fish Audio S2?

Fish Audio S2 represents a groundbreaking leap in voice AI, establishing itself as the most expressive and capable open-source text-to-speech (TTS) model available today. Engineered from the ground up with a focus on expressiveness, speed, and complete openness, S2 empowers developers and creators to generate unbelievably realistic speech with fine-grained control over every nuance.

Unlike traditional TTS systems, S2 is built for dynamic, real-time interaction. Its ultra-low latency, under 150ms, unlocks possibilities for seamless conversational AI, live dubbing, and interactive voice experiences that feel natural and immediate. The model's open-source nature means full access to inference code and model weights, allowing for self-hosting, custom fine-tuning, and integration without vendor lock-in, fostering a community-driven approach to innovation in voice technology.

Key Features

• Unmatched Expressiveness: Control emotions, paralanguage, and subtle vocal inflections with natural text instructions. Generate speech with laughter, whispers, sighs, and more, creating truly lifelike vocal performances.
• Ultra-Low Latency: Achieve response times under 150ms, enabling real-time conversational AI, live dubbing, and interactive applications without compromising quality.
• Open Domain Control & Multi-Speaker: Seamlessly manage speaker transitions within a single generation and control expressive elements using natural language prompts, offering unparalleled flexibility.
• 80+ Language Support: Generate high-quality speech across a vast array of languages, with Tier 1 support for English, Japanese, and Chinese, and robust support for many others.
• Fully Open-Source: Access both inference code and model weights. Run, fine-tune, and integrate S2 on your own infrastructure, ensuring transparency and freedom from vendor lock-in.
• Production-Ready Performance: Optimized with SGLang, S2 offers exceptional speed and efficiency, including features like continuous batching and paged KV cache for high-throughput applications.
• Fine-Grained Inline Control: Embed natural-language instructions directly within text using a flexible tag syntax (e.g., [whisper in small voice], [professional broadcast tone]) for word-level expression control.

How to Use Fish Audio S2

Getting started with Fish Audio S2 is straightforward, whether you're integrating it via API or running it locally.

1. Installation: Install the necessary libraries using pip: pip install fish-audio.
2. API Integration: Initialize the FishAudio client with your API key: client = FishAudio(api_key="your_api_key_here").
3. Speech Generation: Use the client.tts.convert() method, specifying your text, desired model (e.g., s2-pro), and any control tags for expressiveness. For example: audio = client.tts.convert(text="[excited] Hello there! [pause] How can I help you today?", model="s2-pro").
4. Saving Audio: Save the generated audio to a file using a utility function: save(audio, "output.mp3").
5. Local Deployment (Optional): For full control, download the model weights and inference code. Follow the provided documentation to set up the SGLang-based streaming inference engine on your own hardware.

Experiment with different control tags and multi-speaker configurations to achieve the exact vocal performance you need.

Use Cases

Fish Audio S2's advanced capabilities make it ideal for a wide range of applications:

• Conversational AI & Chatbots: Create highly engaging and natural-sounding virtual assistants and chatbots that can convey emotion and personality, leading to better user experiences.
• Gaming & Virtual Worlds: Develop immersive gaming experiences with dynamic NPC dialogue that reacts realistically to in-game events and player interactions.
• Content Creation & Dubbing: Produce professional-quality voiceovers, podcasts, and audiobooks with realistic intonation and emotion. Enable real-time dubbing for videos and live streams with minimal latency.
• Accessibility Tools: Build advanced text-to-speech applications for visually impaired users or those with communication difficulties, offering a more natural and understandable voice output.
• Interactive Voice Response (IVR) Systems: Enhance customer service IVR systems with more human-like and expressive voice prompts, improving caller satisfaction.

FAQ

What is Fish Audio S2 Pro? Fish Audio S2 Pro is an advanced text-to-speech model renowned for its fine-grained control over prosody and emotion. It leverages a Dual-Autoregressive architecture and extensive training data across 80+ languages to deliver highly realistic speech. The release includes model weights, fine-tuning code, and an optimized inference engine.

How does the fine-grained inline control work? S2 Pro allows for localized speech control by embedding natural-language instructions directly within the text using a tag-like syntax (e.g., [pitch up], [laughing]). This enables open-ended expression control at the word level, supporting over 15,000 unique descriptive tags for nuanced vocal performance.

What are the performance metrics for S2 Pro? On high-end GPUs, S2 Pro achieves a Real-Time Factor (RTF) below 0.5, with time-to-first-audio around 100ms. Its SGLang-based inference engine is highly optimized for throughput and low latency, supporting advanced serving techniques.

What is the licensing for Fish Audio S2? Fish Audio S2 is available under the Fish Audio Research License. Research and non-commercial use are free. For commercial use, a separate license is required; please contact [email protected] for details.

How many languages does S2 Pro support? S2 Pro supports over 80 languages, with top-tier quality for English, Japanese, and Chinese. It also offers strong support for languages like Korean, Spanish, Portuguese, Arabic, Russian, French, and German, among many others.