Benchmarks
Speech-to-Text (ASR) Leaderboard
Word error rate and streaming latency for Smallest Pulse against Deepgram, AssemblyAI, ElevenLabs and others, with full methodology and reproducible scripts.
Last updated
16 Oct 2026
Section 01
Latency comparison
Plotted on a shared millisecond scale so the real spread is visible. Averages hide the calls that feel broken, so percentiles appear wherever they exist.
Speech-to-text, time to first transcript
Sorted fastest first. Solid marker is a point measurement, capsule is a P50 to P90 span.
Speechmatics Flow publishes only a sub-1 second figure, too coarse to plot honestly.
Under 150 ms
150 to 300 ms
Over 300 ms
Smallest measured
Speko
Our own numbers disagree. Internal sources give Pulse time to first transcript as both 64 ms and 70 ms, and Deepgram Nova 3 as both 71 ms and 100 ms. Lightning appears as 100 ms end-to-end on our comparison pages and ~200 ms time to first byte in the docs, and end-to-end cannot be faster than its own first byte. The conservative figure is plotted for us and the favourable one for competitors until a single run settles it.
Why percentiles, not means
A voice agent is judged on its worst turns. A model with a 200 ms mean and a 2 second P99 feels broken more often than one with a 300 ms mean and a 400 ms P99, and a mean cannot tell you which one you have. We publish percentiles where we have them and mark the gap where we do not.
Speech-to-text, time to first transcript
Section 02
How we benchmark
The parameters stated up front rather than hidden behind a disclosure. Public benchmarks can be overfitted and gamed, which is an argument for showing your method, not for hiding behind one number.
Datasets
ESB suite, 9 datasets, English. FLEURS for multilingual, 26 languages. WildASR for degraded recording conditions. Internal English and Hindi perturbation suites. EmergentTTS, 1,088 samples drawn from the 1,645-sample public set. AIEWF, 300 tasks over 10 runs of 30 turns. Plus 44 real Hindi-English collections calls.
Accuracy metric
Word error rate under the Whisper normalizer, with character error rate alongside. This choice is material: a third-party board reported Pulse at 8.0% where inverse text normalisation scored spelled-out numbers as errors, against 5.1% otherwise. On numbers-heavy call audio the normaliser changes the answer.
Latency metric
Time to first transcript runs from end of speech to complete transcript. Time to first byte is the first audio byte received. Voice-to-voice is end of user speech to start of agent audio. Reported as P50, P90, P95 and P99 wherever the run produced them.
Ground truth
Manual human transcription for the call benchmark. Public datasets use their own published references, unmodified.
Hardware
Throughput on a single L40S, 48 GB. Streaming tests at 16 kHz linear16. Competitor models called through their public production APIs, not self-hosted.
Refresh cadence
Quarterly, and immediately when any model in the comparison set ships a new version. Every table carries its own measurement date.
Reproduce it
The evaluation walkthrough and scripts are public, and every result below can be regenerated against your own audio.
Section 03
Speech-to-text
Switch regimes to see the same models on clean benchmark audio and on real call recordings.
Clean benchmarks
Real call audio
ESB suite word error rate by domain. Lower is better, green marks the best model in each row.
Domain
Audiobook, clean
Audiobook, noisy
Crowdsourced
Parliament
TED talks
Podcasts
Financial
Earnings calls
Meetings
Overall
Dataset
LibriSpeech Clean
LibriSpeech Other
Common Voice
VoxPopuli
TED-LIUM
GigaSpeech
SPGISpeech
Earnings22
AMI
Aggregate
Pulse
2.46
5.31
10.89
7.16
4.07
10.43
2.86
12.25
10.58
7.33
AssemblyAI
1.65
2.86
6.73
7.28
2.95
9.12
1.74
11.52
14.60
6.49
Deepgram
3.20
6.60
14.22
9.55
3.59
10.05
2.99
15.79
17.04
9.23
ElevenLabs
1.97
4.45
9.83
7.91
3.16
9.66
4.40
12.20
12.23
7.31
Smallest's Pulse achievement. Pulse wins parliamentary speech and meetings, the two hardest multi-speaker conditions. Note also that Pulse streams at 7.33 aggregate here, while the 5.42% headline is Pulse Pro on the pre-recorded path. They are different models. While Assembly AI leads 7 of 9 domains and the aggregate, and ElevenLabs Scribe edges Pulse overall by 0.02.
FLEURS streaming word error rate by language, Pulse against Deepgram Nova 3
LANGUAGE
Italian
Spanish
English
Portuguese
Hindi
Germen
French
Marathi
Gujarati
PULSE
4.41
5.99
6.03
8.32
8.30
9.50
10.71
15.68
20.05
Deepgram NOVA 3
6.99
7.55
11.21
11.46
15.46
10.15
12.07
not supported
not supported
RATIO
1.6x
1.3x
1.9x
1.4x
1.19
1.1x
1.1x
Marathi and Gujarati have no Deepgram column because the model does not cover them. Coverage is itself a result.
Smallest measured
WildASR robustness, word error rate under degraded recording conditions
CONDITION
Clean Audio
Accented speech
Phone codec
Background noise
Reverberation
Far-field mic
Clipping
Overall
Pulse
5.98
5.82
7.19
8.90
9.06
13.38
14.03
9.63
AssemblyAI
3.33
2.80
3.45
4.04
23.50
26.07
6.59
12.52
Deepgram
11.62
7.31
9.13
15.04
27.27
62.99
43.35
28.17
ElevenLabs
4.24
4.01
4.98
6.30
6.48
7.38
11.20
6.74
ElevenLabs Scribe is the most robust model here overall. The useful finding is about failure modes: AssemblyAI is the most accurate on clean audio yet degrades to 26.07 far-field and 23.50 on reverberation, and Deepgram reaches 62.99 far-field. Pulse loses most single conditions but never collapses on any of them, which is what matters when you cannot control how your callers are recorded.
Smallest measured
Pulse: 21 streaming languages, 26 pre-recorded.
Pulse Pro: English only, HTTP only, streaming worker on roadmap. Throughput 250-300x real time on 1x L40S.
Pricing: our own materials quote $0.003, $0.004, $0.005 and $0.008 per minute for real-time Pulse. Not published here until reconciled.
Third-party result: Cekura
Cekura measured Smallest Pulse against 18 other streaming speech-to-text models, last updated Sep 16, 2026. On real phone call audio (Ocular dataset) Pulse ranks 3rd of 19 with 3.46% word error rate. On clean clips (Pipecat dataset) it ranks 15th of 19 with 3.66%. Time to first text at P50 is 1.06 s, 5th of 17, with no failed requests across 1,000 clips. This was measured by Cekura, not by Smallest.
Section 04
Public benchmarks vs your actual audio
Our Hindi word error rate is 8.30% on FLEURS and 29.1% on real collections calls. Both are measured. Neither is wrong.
Same model, same language, two kinds of audio
Word error rate. Lower is better.
Read-speech benchmarks measure clean, scripted audio recorded in good conditions. Production calls are code-switched, noisy, spontaneous and compressed by telephony codecs. The gap is roughly 3.5× for us and 3.5× for Sarvam, so it is a property of the audio rather than of any one model.
We publish both because a single number would mislead you. Whichever figure a vendor quotes, it is probably the easier one, and the only number that predicts your production accuracy is the one you measure on your own recordings.
Section 05
Real Hindi-English call audio
Five providers on 44 real collections calls, scored against manual human transcripts. We win two of six categories. The other four are published unchanged.
Green marks the category winner. Recall scores run 0 to 1, higher is better. July 2026.
Dataset
SMALLEST AI
PULSE
DEEPGRAM
NOVA 3
SERVAM AI
Saaras
ELEVENLABS
Scribe, KEYTERMS
ELEVENLABS
Scribe, standard
WER
0.291
0.325
0.376
0.460
0.554
LOAN TERMS
0.872
0.919
0.910
0.902
0.815
PROMISE TO PAY
0.557
0.509
0.502
0.432
0.381
NUMBERS
0.387
0.282
0.381
0.410
0.389
AMOUNT
0.334
0.280
0.330
0.349
0.313
DATES
0.531
0.510
0.522
0.611
0.485
What to take from this
Pulse gives the most accurate and most consistent transcript. Deepgram is strongest on loan terminology. ElevenLabs Keyterms captures financial entities best but produces a noticeably worse transcript around them. Always choose Keyterms over standard ElevenLabs; it wins every metric at the same speed and cost.
An industry-wide gap
Across all five providers, amounts and numbers recall stayed low and none cleared 0.35. That is an unsolved problem for the whole category rather than a gap in any one product, and it means any amount-critical workflow needs a verification step whichever API you choose.
Scope and limits. 44 recordings, July 2026, Hindi-English collections and loan-servicing calls from a single domain. Ground truth is manual human transcription. The dataset carries no noise-condition labels and no language-mix breakdown, so nothing here supports a conclusion about specific noise levels or code-switching ratios. At n=44, small gaps between adjacent models may not be significant.
Source data and further reading
Section 06
Entity-level recall
Word error rate treats every word as equal. In a collections call, one missed rupee figure matters more than ten missed filler words. No public leaderboard measures this, so we defined it.
Loan terms
Did the model capture loan vocabulary such as EMI, tenure and interest.
Promise to pay
Did it capture commitment phrases such as "I will pay by" or "will transfer today".
Numbers
Did it correctly transcribe spoken numeric values.
Amounts
Did it capture spoken rupee figures.
Scoring
All scored 0 to 1 and averaged over 44 recordings. These are Smallest-defined metrics. The term lists and matching logic still need publishing before an outside team can reproduce them.
Promise-to-pay recall
Higher is better, on the full 0 to 1 scale so the true size of the gaps is visible.
Amounts recall
The dashed line marks 0.35. No provider clears it.
Section 07
FAQ
What is word error rate?
How is speech-to-text latency measured?
Why do your two Hindi numbers differ so much?
Which model should I use for noisy call audio?
Do you support Hindi and code-switching?
Are these numbers independent?