Benchmarks
Text-to-Speech Leaderboard
Listener preference, quality scores and time to first audio for Smallest Lightning against ElevenLabs, OpenAI, Cartesia and Murf, with full methodology and reproducible scripts.
Last updated
16 Oct 2026
Section 01
Latency comparison
Plotted on a shared millisecond scale so the real spread is visible. Averages hide the calls that feel broken, so percentiles appear wherever they exist.
Text-to-speech, time to first audio
Sorted fastest first. Vendor ranges are drawn as capsules across their whole span, not collapsed to a midpoint.
Murf and Cartesia are faster than Lightning on these figures. Cartesia's is a 90th percentile and WellSaid's is per 30 characters, so the set is not measured like for like.
Under 200 ms
200 to 300 ms
Over 400 ms
Smallest measured
Why percentiles, not means
A voice agent is judged on its worst turns. A model with a 200 ms mean and a 2 second P99 feels broken more often than one with a 300 ms mean and a 400 ms P99, and a mean cannot tell you which one you have. We publish percentiles where we have them and mark the gap where we do not.
Text-to-speech, time to first audio
Section 02
How we benchmark
The parameters stated up front rather than hidden behind a disclosure. Public benchmarks can be overfitted and gamed, which is an argument for showing your method, not for hiding behind one number.
Datasets
ESB suite, 9 datasets, English. FLEURS for multilingual, 26 languages. WildASR for degraded recording conditions. Internal English and Hindi perturbation suites. EmergentTTS, 1,088 samples drawn from the 1,645-sample public set. AIEWF, 300 tasks over 10 runs of 30 turns. Plus 44 real Hindi-English collections calls.
Accuracy metric
Word error rate under the Whisper normalizer, with character error rate alongside. This choice is material: a third-party board reported Pulse at 8.0% where inverse text normalisation scored spelled-out numbers as errors, against 5.1% otherwise. On numbers-heavy call audio the normaliser changes the answer.
Latency metric
Time to first transcript runs from end of speech to complete transcript. Time to first byte is the first audio byte received. Voice-to-voice is end of user speech to start of agent audio. Reported as P50, P90, P95 and P99 wherever the run produced them.
Ground truth
Manual human transcription for the call benchmark. Public datasets use their own published references, unmodified.
Hardware
Throughput on a single L40S, 48 GB. Streaming tests at 16 kHz linear16. Competitor models called through their public production APIs, not self-hosted.
Refresh cadence
Quarterly, and immediately when any model in the comparison set ships a new version. Every table carries its own measurement date.
Reproduce it
The evaluation walkthrough and scripts are public, and every result below can be regenerated against your own audio.
Section 03
Text-to-speech
Blind head-to-head listener preference on the EmergentTTS benchmark, then the same models rated dimension by dimension.
Lightning win rate against each competitor
1,088 samples. The solid line is 50%, where neither model is preferred.
GPT-4o-mini TTS beats Lightning on this benchmark. That row stays in. The 1,088 samples are a subset of the 1,645-sample public EmergentTTS set.
Smallest measured
Naturalness
Metric
Overall
Naturalness
Intonation
Prosody
Lightning Pro
3.16
2.55
3.06
2.81
GPT-4o-mini
3.13
2.41
3.06
2.73
EL Turbo v2.5
3.16
2.52
3.07
2.73
Sonic-3
3.20
2.57
3.12
2.83
EXPRESSIVENESS
Overall
Paralinguistics
Emotions
3.55
3.64
3.47
3.45
3.60
3.30
3.44
3.59
3.28
3.38
3.56
2.83
MOS V2
Mean MOS
UTMOS, predicted
WV-MOS, predicted
4.22
3.77
5.05
4.16
3.76
4.55
3.98
3.37
4.60
3.76
2.77
4.76
DELIVERY
Natural pace
Pause placement
Breathing naturalness
Pronunciation style
Boundary consistency
4.72
4.66
3.82
4.98
4.96
4.57
4.54
3.06
4.96
4.94
4.51
4.49
3.14
4.95
4.93
4.01
4.28
2.79
4.96
4.93
Performance Lightning's advantage is expressiveness, delivery and breathing, where the gap is large: 3.82 against 2.79 on breathing naturalness. This also contradicts the head-to-head chart above, where Lightning was preferred over Sonic-3 68.3% of the time. Blind preference and dimensional rating measure different things, and both are reported rather than only the flattering one. UTMOS and WV-MOS are model-predicted, not human ratings.
Synthesis accuracy, transcribed back with Whisper and scored against the input text
Metric
Lightning v3.1
Lightning v3.1 Pro
wer
1.57%
1.36%
cer
0.67%
0.40%
hallucination
0.03%
0.00%
15 languages including Hindi, Tamil, Kannada, Telugu, Malayalam, Marathi and Gujarati, with automatic language detection and mid-sentence switching.
Time to first byte ~200 ms at 40 concurrent streams.
Pricing: three different figures appear across our own materials for the same product, $0.05, $0.145 and $0.175 per 10K characters. Not published here until reconciled.
Source data and further reading
Section 04
FAQ
How do you measure text-to-speech quality?
Does Lightning win every comparison?
What is the difference between TTFB, TTFT and voice-to-voice?
Which languages does Lightning support?
Are these numbers independent?
How often are these benchmarks refreshed?