Benchmarks
Speech-to-Speech & Voice Agent Benchmarks
Voice-to-voice latency and task accuracy for Smallest Hydra against other production speech-to-speech models, with full methodology.
Last updated
16 Oct 2026
Section 01
How we benchmark
The parameters stated up front rather than hidden behind a disclosure. Public benchmarks can be overfitted and gamed, which is an argument for showing your method, not for hiding behind one number.
Datasets
ESB suite, 9 datasets, English. FLEURS for multilingual, 26 languages. WildASR for degraded recording conditions. Internal English and Hindi perturbation suites. EmergentTTS, 1,088 samples drawn from the 1,645-sample public set. AIEWF, 300 tasks over 10 runs of 30 turns. Plus 44 real Hindi-English collections calls.
Accuracy metric
Word error rate under the Whisper normalizer, with character error rate alongside. This choice is material: a third-party board reported Pulse at 8.0% where inverse text normalisation scored spelled-out numbers as errors, against 5.1% otherwise. On numbers-heavy call audio the normaliser changes the answer.
Latency metric
Time to first transcript runs from end of speech to complete transcript. Time to first byte is the first audio byte received. Voice-to-voice is end of user speech to start of agent audio. Reported as P50, P90, P95 and P99 wherever the run produced them.
Ground truth
Manual human transcription for the call benchmark. Public datasets use their own published references, unmodified.
Hardware
Throughput on a single L40S, 48 GB. Streaming tests at 16 kHz linear16. Competitor models called through their public production APIs, not self-hosted.
Refresh cadence
Quarterly, and immediately when any model in the comparison set ships a new version. Every table carries its own measurement date.
Reproduce it
The evaluation walkthrough and scripts are public, and every result below can be regenerated against your own audio.
Section 02: Beta
Speech-to-speech
Voice-to-voice latency across 9 production models on the AIEWF benchmark: 300 tasks, 10 runs of 30 turns
Tool-call voice-to-voice latency, mean
Lower is better. The dashed line marks 2 seconds, where a turn starts to feel slow.
Non-tool voice-to-voice median is 864 ms, tied fastest of the set. Percentile distributions are not yet published.
Smallest measured
Task performance, AIEWF pass rate and Audio MultiChallenge
MODEL
Hydra
Ultravox v0.7
GPT Realtime 1.5
GPT Realtime
GPT Realtime 2
WER
95.9
97.7%
93.3%
86.7%
not run
Audio multichallenge
35.52%
not run
34.73%
not run
37.61%
Source data and further reading
Section 03
FAQ
How is voice-to-voice latency measured?
What is the difference between TTFB, TTFT and voice-to-voice?
Why is speech-to-speech marked beta?
Are these numbers independent?
How often are these benchmarks refreshed?
Can I reproduce these results?