Voice AI BenchmarksBy

DATASETSTTWILDASR / FLEURSACTIVE

manifest

WildASR noise gaps speech-to-text benchmark dataset

284 clips with intermittent noise bursts and silence segments inserted, so every clip runs longer than its clean sibling.

Items
284
fixed public inputs
Models measured
31
last 30 days
Current leader · WER#1 / 31
3.1%
Universal 3.5 Provia AssemblyAI

How models rank on WildASR noise gaps

Full STT dashboard
Word Error Rate on WildASR noise gapsPercent · lower is better · 30-day average on this datasetEvery STT model measured on the WildASR noise gaps dataset, ranked on Word Error Rate.
  1. #1Universal 3.5 Pro3.1%
  2. #2Universal 3.6 Pro3.2%
  3. #3GPT-4o Transcribe3.7%
  4. #4Qwen3 ASR Fast3.7%
  5. #5resonant-13.8%
  6. #6Qwen3 ASR 1.7b4.2%
  7. #7Gemini 3.5 Transcribe Live4.4%
  8. #8Chirp 34.4%
  9. #9GPT-4o mini Transcribe4.9%
  10. #10Ink 25.4%
  11. #11Chirp 25.5%
  12. #12Grok STT5.6%
Show all 31 models
  1. #13GPT Realtime Whisper5.8%
  2. #14Scribe v2 Realtime6.1%
  3. #15Voxtral Mini Transcribe Realtime 26026.2%
  4. #16Whisper Large v3via Baseten6.4%
  5. #17Pulse6.6%
  6. #18Linden 16.6%
  7. #19STT RT v57.1%
  8. #20STT 18.2%
  9. #21Nova 38.4%
  10. #22Solaria 18.5%
  11. #23Universal Streaming8.5%
  12. #24Flux9.0%
  13. #25Daytona STT9.9%
  14. #26Flux Multilingual10.5%
  15. #27Nova 210.6%
  16. #28Whisper Large v3via Together AI10.8%
  17. #29Default12.5%
  18. #30Nemotron 3.5 ASR Streaming15.7%
  19. #31Parakeet TDT 0.6B v316.4%
median of all models · 6.4%
STT models on the WildASR noise gaps dataset over the last 30 days, ranked on Word Error Rate.
#ModelHostTTFSWERTTFTSamples
1Universal 3.5 ProAssemblyAI186 ms904 ms1,403
2Universal 3.6 ProAssemblyAI132 ms963 ms1,283
3GPT-4o TranscribeOpenAI795 ms—1,402
4Qwen3 ASR FastNari Labs45 ms1750 ms1,402
5resonant-1Reson8251 ms—1,403
6Qwen3 ASR 1.7bBaseten39 ms863 ms720
7Gemini 3.5 Transcribe LiveGemini319 ms1600 ms1,401
8Chirp 3Google782 ms6269 ms1,402
9GPT-4o mini TranscribeOpenAI713 ms—1,401
10Ink 2Cartesia149 ms1927 ms1,403
11Chirp 2Google884 ms6400 ms1,403
12Grok STTSpaceXAI217 ms—1,397
13GPT Realtime WhisperOpenAI602 ms1787 ms1,401
14Scribe v2 RealtimeElevenLabs123 ms2127 ms1,402
15Voxtral Mini Transcribe Realtime 2602Mistral402 ms1819 ms1,380
16Whisper Large v3Baseten116 ms835 ms720
17PulseSmallest205 ms1910 ms1,403
18Linden 1Speechmatics283 ms1497 ms1,396
19STT RT v5Soniox44 ms1456 ms1,239
20STT 1Inworld AI70 ms1167 ms1,304
21Nova 3Deepgram86 ms1223 ms1,369
22Solaria 1Gladia582 ms1652 ms1,383
23Universal StreamingAssemblyAI—1424 ms1,401
24FluxDeepgram100 ms930 ms1,369
25Daytona STTGuava382 ms4570 ms288
26Flux MultilingualDeepgram103 ms1012 ms1,369
27Nova 2Deepgram87 ms1204 ms1,369
28Whisper Large v3Together AI173 ms1195 ms1,391
29DefaultGradium278 ms1969 ms1,402
30Nemotron 3.5 ASR StreamingTogether AI—1429 ms1,401
31Parakeet TDT 0.6B v3Together AI92 ms1142 ms1,394
Under-sampled models are excluded; tied models share a place. Dotted WER values split into substitutions, deletions and insertions on hover or tap.

Inside the dataset

284 fixed public inputs — every model is tested on exactly these.

  • CLIP 1 · 13.6s

    “"he [wales] basically lied to us from the start. first by acting as if this was for legal reasons. second by pretending he was listening to us right up to his art deletion.”

  • CLIP 2 · 11.2s

    “a car bomb detonated at police headquarters in gaziantep turkey yesterday morning killed two police officers and injured more than twenty other people”

  • CLIP 3 · 9.5s

    “a civilization is a singular culture shared by a significant large group of people who live and work co-operatively a society”

3 of 284 clips, streamed from the public benchmark storage bucket.

Source: bosonai/WildASR environment_degradation__en__fleurs_noise_gap_en · License: Apache-2.0 (WildASR; audio derived from FLEURS, CC-BY-4.0)

WER change from the clean baseline

Same provider and model on this condition vs the WildASR clean dataset; positive = more recognition error.

WildASR noise gaps vs WildASR cleanPercentage points of WER · diverging from no changePer-model WER change between WildASR clean and WildASR noise gaps.
zero = no change from the clean baseline · right = more error under this condition
Per-model WER change between WildASR clean and WildASR noise gaps.
ModelHostClean WERCondition WERChange
Universal 3.5 ProAssemblyAI2.8%3.1%+0.3%
Universal 3.6 ProAssemblyAI2.5%3.2%+0.7%
GPT-4o TranscribeOpenAI2.8%3.7%+0.8%
Qwen3 ASR FastNari Labs3.5%3.7%+0.2%
resonant-1Reson83.2%3.8%+0.6%
Qwen3 ASR 1.7bBaseten3.7%4.2%+0.5%
Gemini 3.5 Transcribe LiveGemini3.1%4.4%+1.3%
Chirp 3Google3.4%4.4%+1.0%
GPT-4o mini TranscribeOpenAI3.2%4.9%+1.7%
Ink 2Cartesia4.5%5.4%+0.9%
Chirp 2Google4.6%5.5%+1.0%
Grok STTSpaceXAI4.1%5.6%+1.5%
GPT Realtime WhisperOpenAI4.1%5.8%+1.7%
Scribe v2 RealtimeElevenLabs4.0%6.1%+2.1%
Voxtral Mini Transcribe Realtime 2602Mistral4.6%6.2%+1.5%
Whisper Large v3Baseten4.3%6.4%+2.2%
PulseSmallest5.1%6.6%+1.5%
Linden 1Speechmatics4.8%6.6%+1.8%
STT RT v5Soniox6.1%7.1%+1.0%
STT 1Inworld AI8.1%8.2%+0.2%
Nova 3Deepgram6.1%8.4%+2.3%
Solaria 1Gladia6.0%8.5%+2.5%
Universal StreamingAssemblyAI6.1%8.5%+2.4%
FluxDeepgram5.4%9.0%+3.6%
Daytona STTGuava5.2%9.9%+4.8%
Flux MultilingualDeepgram7.2%10.5%+3.3%
Nova 2Deepgram6.9%10.6%+3.8%
Whisper Large v3Together AI6.7%10.8%+4.1%
DefaultGradium7.8%12.5%+4.7%
Nemotron 3.5 ASR StreamingTogether AI10.5%15.7%+5.3%
Parakeet TDT 0.6B v3Together AI10.7%16.4%+5.7%

What it tests

This condition inserts noise bursts and silent gaps into each clean clip. It measures whether transcription recovers after an interruption and also tests longer audio sequences than the paired baseline.

This condition is derived from the WildASR environment family and FLEURS read speech. Each paired condition uses the same utterance and transcript as the WildASR clean baseline, so the WER change isolates the effect of the test condition.

Metrics reported for this dataset

Same datasets, prompts and metric definitions for every model, measured by Coval’s open-source runner. Full methodology on the overview.

Evaluate your own voice agent

Use Coval to test your production configuration, prompts and calls—not only the public benchmark endpoints.

Book a Demo