Voice AI BenchmarksBy

DATASETSTTWILDASR / FLEURSACTIVE

manifest

WildASR phone codec speech-to-text benchmark dataset

284 clips passed through telephony compression, with two codec conditions rotated across the pool.

Items
284
fixed public inputs
Models measured
31
last 30 days
Current leader · WER#1 / 31
2.6%
Universal 3.6 Provia AssemblyAI

How models rank on WildASR phone codec

Full STT dashboard
Word Error Rate on WildASR phone codecPercent · lower is better · 30-day average on this datasetEvery STT model measured on the WildASR phone codec dataset, ranked on Word Error Rate.
  1. #1Universal 3.6 Pro2.6%
  2. #2Universal 3.5 Pro2.8%
  3. #3GPT-4o Transcribe3.1%
  4. #4Chirp 33.5%
  5. #5Qwen3 ASR Fast3.6%
  6. #6resonant-13.6%
  7. #7GPT-4o mini Transcribe3.9%
  8. #8Gemini 3.5 Transcribe Live4.5%
  9. #9Qwen3 ASR 1.7b5.0%
  10. #10Ink 25.0%
  11. #11Linden 15.2%
  12. #12Scribe v2 Realtime5.3%
Show all 31 models
  1. #13Chirp 25.5%
  2. #14Whisper Large v3via Baseten5.6%
  3. #15GPT Realtime Whisper5.6%
  4. #16Daytona STT5.7%
  5. #17Voxtral Mini Transcribe Realtime 26026.0%
  6. #18Pulse6.4%
  7. #19STT 16.6%
  8. #20Grok STT6.6%
  9. #21STT RT v56.8%
  10. #22Nova 37.2%
  11. #23Universal Streaming7.4%
  12. #24Flux7.9%
  13. #25Solaria 18.1%
  14. #26Whisper Large v3via Together AI8.8%
  15. #27Parakeet TDT 0.6B v39.0%
  16. #28Nova 29.2%
  17. #29Flux Multilingual9.6%
  18. #30Default10.4%
  19. #31Nemotron 3.5 ASR Streaming13.7%
median of all models · 5.7%
STT models on the WildASR phone codec dataset over the last 30 days, ranked on Word Error Rate.
#ModelHostTTFSWERTTFTSamples
1Universal 3.6 ProAssemblyAI136 ms986 ms1,273
2Universal 3.5 ProAssemblyAI178 ms956 ms1,405
3GPT-4o TranscribeOpenAI792 ms—1,402
4Chirp 3Google787 ms6329 ms1,403
5Qwen3 ASR FastNari Labs47 ms1751 ms1,405
6resonant-1Reson8251 ms—1,405
7GPT-4o mini TranscribeOpenAI731 ms—1,402
8Gemini 3.5 Transcribe LiveGemini315 ms1742 ms1,403
9Qwen3 ASR 1.7bBaseten27 ms923 ms699
10Ink 2Cartesia153 ms1921 ms1,405
11Linden 1Speechmatics293 ms1520 ms1,400
12Scribe v2 RealtimeElevenLabs123 ms2122 ms1,404
13Chirp 2Google938 ms6503 ms1,405
14Whisper Large v3Baseten131 ms845 ms699
15GPT Realtime WhisperOpenAI631 ms1794 ms1,403
16Daytona STTGuava387 ms5704 ms264
17Voxtral Mini Transcribe Realtime 2602Mistral427 ms1797 ms1,380
18PulseSmallest218 ms1973 ms1,405
19STT 1Inworld AI66 ms1223 ms1,303
20Grok STTSpaceXAI195 ms—1,402
21STT RT v5Soniox44 ms1442 ms1,241
22Nova 3Deepgram91 ms1214 ms1,371
23Universal StreamingAssemblyAI—1518 ms1,402
24FluxDeepgram108 ms1040 ms1,371
25Solaria 1Gladia609 ms1740 ms1,380
26Whisper Large v3Together AI197 ms1209 ms1,393
27Parakeet TDT 0.6B v3Together AI91 ms1163 ms1,397
28Nova 2Deepgram94 ms1234 ms1,371
29Flux MultilingualDeepgram105 ms1026 ms1,371
30DefaultGradium299 ms1961 ms1,404
31Nemotron 3.5 ASR StreamingTogether AI—1406 ms1,400
Under-sampled models are excluded; tied models share a place. Dotted WER values split into substitutions, deletions and insertions on hover or tap.

Inside the dataset

284 fixed public inputs — every model is tested on exactly these.

  • CLIP 1 · 12.4s

    “"he [wales] basically lied to us from the start. first by acting as if this was for legal reasons. second by pretending he was listening to us right up to his art deletion.”

  • CLIP 2 · 10.0s

    “a car bomb detonated at police headquarters in gaziantep turkey yesterday morning killed two police officers and injured more than twenty other people”

  • CLIP 3 · 8.9s

    “a civilization is a singular culture shared by a significant large group of people who live and work co-operatively a society”

3 of 284 clips, streamed from the public benchmark storage bucket.

Source: bosonai/WildASR environment_degradation__en__fleurs_phone_codec_en · License: Apache-2.0 (WildASR; audio derived from FLEURS, CC-BY-4.0)

WER change from the clean baseline

Same provider and model on this condition vs the WildASR clean dataset; positive = more recognition error.

WildASR phone codec vs WildASR cleanPercentage points of WER · diverging from no changePer-model WER change between WildASR clean and WildASR phone codec.
zero = no change from the clean baseline · right = more error under this condition
Per-model WER change between WildASR clean and WildASR phone codec.
ModelHostClean WERCondition WERChange
Universal 3.6 ProAssemblyAI2.5%2.6%+0.1%
Universal 3.5 ProAssemblyAI2.8%2.8%+0.0%
GPT-4o TranscribeOpenAI2.8%3.1%+0.3%
Chirp 3Google3.4%3.5%+0.1%
Qwen3 ASR FastNari Labs3.5%3.6%+0.1%
resonant-1Reson83.2%3.6%+0.4%
GPT-4o mini TranscribeOpenAI3.2%3.9%+0.7%
Gemini 3.5 Transcribe LiveGemini3.1%4.5%+1.5%
Qwen3 ASR 1.7bBaseten3.7%5.0%+1.3%
Ink 2Cartesia4.5%5.0%+0.5%
Linden 1Speechmatics4.7%5.2%+0.4%
Scribe v2 RealtimeElevenLabs4.0%5.3%+1.3%
Chirp 2Google4.6%5.5%+0.9%
Whisper Large v3Baseten4.3%5.6%+1.4%
GPT Realtime WhisperOpenAI4.1%5.6%+1.5%
Daytona STTGuava5.0%5.7%+0.6%
Voxtral Mini Transcribe Realtime 2602Mistral4.6%6.0%+1.3%
PulseSmallest5.1%6.4%+1.4%
STT 1Inworld AI8.0%6.6%-1.5%
Grok STTSpaceXAI4.1%6.6%+2.6%
STT RT v5Soniox6.1%6.8%+0.7%
Nova 3Deepgram6.1%7.2%+1.1%
Universal StreamingAssemblyAI6.1%7.4%+1.3%
FluxDeepgram5.4%7.9%+2.5%
Solaria 1Gladia6.1%8.1%+2.0%
Whisper Large v3Together AI6.7%8.8%+2.1%
Parakeet TDT 0.6B v3Together AI10.7%9.0%-1.7%
Nova 2Deepgram6.9%9.2%+2.3%
Flux MultilingualDeepgram7.2%9.6%+2.4%
DefaultGradium7.8%10.4%+2.7%
Nemotron 3.5 ASR StreamingTogether AI10.5%13.7%+3.2%

What it tests

This condition passes the clean recordings through common telephony codecs. It measures recognition after the bandwidth limits and compression introduced by a phone network.

This condition is derived from the WildASR environment family and FLEURS read speech. Each paired condition uses the same utterance and transcript as the WildASR clean baseline, so the WER change isolates the effect of the test condition.

Metrics reported for this dataset

Same datasets, prompts and metric definitions for every model, measured by Coval’s open-source runner. Full methodology on the overview.

Evaluate your own voice agent

Use Coval to test your production configuration, prompts and calls—not only the public benchmark endpoints.

Book a Demo