Benchmarks
Voice AI Benchmarks
Results for Pulse, Lightning and Hydra across speech-to-text, text-to-speech and speech-to-speech, with full methodology and reproducible scripts.
Last updated
16 Oct 2026
Section 01
Three numbers, three levels of proof
Every figure on this page carries a badge naming who measured it. Only one of these three is independently verifiable, and we say so.
Pulse Pro · STT
5.42%
Average ESB word error rate
Tied 2nd of 86 systems on the public Open ASR Leaderboard, across 8 datasets.
Lightning v3.1 · TTS
~200ms
Time to first byte
At 40 concurrent requests over WebSocket streaming. Real-time factor 3.3× on one L40S.
Providers compared on this page
Deepgram
Cartesia
OpenAI
AssemblyAI
Murf
Speechmatics
WellSaid
Soniox
Gladia
NVIDIA
ElevenLabs
Sarvam
Explore the full benchmarks
Speech-to-text
Word error rate, latency and real-call accuracy for Pulse. 5.42% average ESB word error rate on Pulse Pro, tied 2nd of 86 on the Open ASR Leaderboard.
Text-to-speech
Listener preference, quality scores and time to first audio for Lightning. About 200 ms to first byte at 40 concurrent requests.
Speech-to-speech
Voice-to-voice latency and task accuracy for Hydra. 1,624 ms on tool calls, fastest of 9 production models.
Section 02
How we benchmark
The parameters stated up front rather than hidden behind a disclosure. Public benchmarks can be overfitted and gamed, which is an argument for showing your method, not for hiding behind one number.
Datasets
ESB suite, 9 datasets, English. FLEURS for multilingual, 26 languages. WildASR for degraded recording conditions. Internal English and Hindi perturbation suites. EmergentTTS, 1,088 samples drawn from the 1,645-sample public set. AIEWF, 300 tasks over 10 runs of 30 turns. Plus 44 real Hindi-English collections calls.
Accuracy metric
Word error rate under the Whisper normalizer, with character error rate alongside. This choice is material: a third-party board reported Pulse at 8.0% where inverse text normalisation scored spelled-out numbers as errors, against 5.1% otherwise. On numbers-heavy call audio the normaliser changes the answer.
Latency metric
Time to first transcript runs from end of speech to complete transcript. Time to first byte is the first audio byte received. Voice-to-voice is end of user speech to start of agent audio. Reported as P50, P90, P95 and P99 wherever the run produced them.
Ground truth
Manual human transcription for the call benchmark. Public datasets use their own published references, unmodified.
Hardware
Throughput on a single L40S, 48 GB. Streaming tests at 16 kHz linear16. Competitor models called through their public production APIs, not self-hosted.
Refresh cadence
Quarterly, and immediately when any model in the comparison set ships a new version. Every table carries its own measurement date.
Reproduce it
The evaluation walkthrough and scripts are public, and every result below can be regenerated against your own audio.
Section 03
Methodology and changelog
Dataset inventory, provenance policy, and the open items we have not resolved.
Dataset inventory
BENCHMARK
ESB suite
FLURS
WildASR
EmergentTTS
AIEWF
Audio MultiChallenge
Hindi-English calls
Perturbation suites
Scope
9 datasets, English
26 languages, read speech
7 degradation conditions
1,088 of 1645 samples
300 tasks, 10x30 turns
Multi-turn audio reasoning
44 collections recordings
English and Hindi
used in
Speech-to-text page
Speech-to-text page
Speech-to-text page
Text-to-speech page
Speech-to-speech page
Speech-to-speech page
Speech-to-text and text-to-speech pages
Speech-to-text page
publicly available
Yes
Yes
Yes
Yes
Yes
Yes
No, Internal
No, Internal
Section 04
FAQ
What is word error rate?
What is the difference between TTFB, TTFT and voice-to-voice?
Are these numbers independent?
Where are the detailed results?
How often are these benchmarks refreshed?
Can I reproduce these results?