Voice AI BenchmarksBy

DATASETSTTPIPECAT STT BENCHMARK DATAACTIVE

manifest

PipeCat (production) speech-to-text benchmark dataset

897 spontaneous voice-agent turns from pipecat-ai/stt-benchmark-data: fragments, fillers and prompted turns, with model-generated reference transcripts.

Items
897
fixed public inputs
Models measured
31
last 30 days
Current leader · WER#1 / 31
1.7%
Gemini 3.5 Transcribe Livevia Gemini

How models rank on PipeCat (production)

Full STT dashboard
Word Error Rate on PipeCat (production)Percent · lower is better · 30-day average on this datasetEvery STT model measured on the PipeCat (production) dataset, ranked on Word Error Rate.
  1. #1Gemini 3.5 Transcribe Live1.7%
  2. #2Universal 3.6 Pro1.7%
  3. #3Qwen3 ASR Fast1.8%
  4. #4resonant-11.8%
  5. #5Universal 3.5 Pro1.9%
  6. #6Qwen3 ASR 1.7b1.9%
  7. #7Chirp 32.0%
  8. #8Ink 22.1%
  9. #9GPT Realtime Whisper2.1%
  10. #10Chirp 22.5%
  11. #11Grok STT2.5%
  12. #12Linden 12.7%
Show all 31 models
  1. #13STT RT v52.8%
  2. #14Daytona STT2.8%
  3. #15Pulse3.2%
  4. #16Nova 33.3%
  5. #17GPT-4o mini Transcribe3.4%
  6. #18Scribe v2 Realtime3.4%
  7. #19Whisper Large v3via Baseten3.7%
  8. #20Flux3.8%
  9. #21GPT-4o Transcribe3.9%
  10. #22Nova 23.9%
  11. #23Universal Streaming3.9%
  12. #24Voxtral Mini Transcribe Realtime 26024.1%
  13. #25Solaria 14.6%
  14. #26Flux Multilingual4.9%
  15. #27Default6.2%
  16. #28STT 16.2%
  17. #29Whisper Large v3via Together AI6.2%
  18. #30Nemotron 3.5 ASR Streaming10.6%
  19. #31Parakeet TDT 0.6B v310.8%
median of all models · 3.3%
STT models on the PipeCat (production) dataset over the last 30 days, ranked on Word Error Rate.
#ModelHostTTFSWERTTFTSamples
1Gemini 3.5 Transcribe LiveGemini321 ms1661 ms13,996
2Universal 3.6 ProAssemblyAI137 ms1206 ms12,777
3Qwen3 ASR FastNari Labs47 ms1751 ms14,019
4resonant-1Reson8254 ms—14,019
5Universal 3.5 ProAssemblyAI163 ms1173 ms14,020
6Qwen3 ASR 1.7bBaseten32 ms1079 ms7,195
7Chirp 3Google780 ms5878 ms14,014
8Ink 2Cartesia147 ms2077 ms14,018
9GPT Realtime WhisperOpenAI592 ms1922 ms14,004
10Chirp 2Google924 ms6007 ms14,017
11Grok STTSpaceXAI226 ms—13,985
12Linden 1Speechmatics264 ms1627 ms14,000
13STT RT v5Soniox45 ms1632 ms12,376
14Daytona STTGuava404 ms5928 ms2,653
15PulseSmallest202 ms2075 ms14,020
16Nova 3Deepgram83 ms1640 ms13,678
17GPT-4o mini TranscribeOpenAI706 ms—14,003
18Scribe v2 RealtimeElevenLabs121 ms2195 ms14,013
19Whisper Large v3Baseten137 ms1048 ms7,192
20FluxDeepgram96 ms1276 ms13,678
21GPT-4o TranscribeOpenAI790 ms—14,006
22Nova 2Deepgram86 ms1601 ms13,679
23Universal StreamingAssemblyAI—1612 ms14,020
24Voxtral Mini Transcribe Realtime 2602Mistral409 ms1962 ms13,807
25Solaria 1Gladia633 ms1914 ms13,790
26Flux MultilingualDeepgram94 ms1334 ms13,680
27DefaultGradium242 ms2097 ms14,010
28STT 1Inworld AI65 ms1259 ms13,015
29Whisper Large v3Together AI202 ms1398 ms13,908
30Nemotron 3.5 ASR StreamingTogether AI—1644 ms13,969
31Parakeet TDT 0.6B v3Together AI99 ms1372 ms13,908
Under-sampled models are excluded; tied models share a place. Dotted WER values split into substitutions, deletions and insertions on hover or tap.

Inside the dataset

897 fixed public inputs — every model is tested on exactly these.

  • CLIP 1 · 10.0s

    “A brand new warning light just appeared on my car's dashboard that I don't recognize. Please help me identify what the symbol means and its level of urgency for service.”

  • CLIP 2 · 11.6s

    “A pop-up window appeared on my computer screen claiming that my system was infected with multiple viruses and that I needed to call a specific phone number immediately for technical support.”

  • CLIP 3 · 6.4s

    “Add a package of whole wheat tortillas and a jar of mild salsa to my list.”

3 of 897 clips, streamed from the public benchmark storage bucket.

Source: pipecat-ai/stt-benchmark-data train · License: unspecified (no data license published by pipecat-ai)

What it tests

This set contains spontaneous turns with hesitations, restarts, fillers and sentence fragments. It tests speech patterns that are not represented in read-speech datasets.

The reference transcripts are model-generated rather than human-verified. Reference errors can increase the absolute WER for every provider, so this dataset is more reliable for comparing models than for interpreting WER as an exact transcription error rate.

Metrics reported for this dataset

Same datasets, prompts and metric definitions for every model, measured by Coval’s open-source runner. Full methodology on the overview.

Evaluate your own voice agent

Use Coval to test your production configuration, prompts and calls—not only the public benchmark endpoints.

Book a Demo