AI
Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Hume has introduced Real World VoiceEQ, a new benchmark designed to evaluate the human quality of voice AI interactions.
Key takeaways
- Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics.
- Traditional benchmarks increasingly overestimate real-world performance of voice AI models.
- Voice models have become better at speaking than actually listening, often ignoring paralinguistic cues like tone and pacing.
- No single system configuration ranked among the top five across all eight capability groups in TTS evaluations.
- Speech-language models (SLMs) should be used more carefully for voice evaluation as they struggle with subjective judgments compared to human raters.
Hume has introduced Real World VoiceEQ, a new benchmark designed to evaluate the human quality of voice AI interactions. The benchmark assesses how well voice systems recognize, produce, and respond to acoustic information that transcripts omit, such as tone, emotion, speaker identity, and background context. It evaluates over 40 proprietary and open-source models across more than 15 dimensions and 60 metrics, utilizing over 1 million human ratings collected via Hume's Kairos evaluation platform.
By the numbers
- 40+
- Leading proprietary and open-source voice models evaluated
- 15+
- Key evaluation dimensions assessed by the benchmark
- 60+
- Metrics spanning ASR, TTS, S2S, and Speech Understanding
- 1M+
- Individual human ratings collected to develop the benchmark
- 4x
- Higher transcription word error rates on noise-backed vs music-backed speech
Turn stories like this into views
Ravenclip finds the AI news, makes the video, and posts it before attention moves on.
Common questions
- What happened with Real World VoiceEQ?
- Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics.
- Where can I read the original report?
- Read the full report at huggingface_blog.
