Voice AI benchmarks in real world conditions.

cleanSTT

Clean read speech — 284 FLEURS clips, the undistorted control

accentSTT

Accented speech — 845 clips spanning demographic accents

noisegapSTT

Background noise — 284 clips with intermittent noise bursts

reverbSTT

Reverberation — 284 clips with room echo

farfieldSTT

Far-field mic — 284 clips recorded at distance

clippingSTT

Clipped audio — 284 clips with peak distortion

phonecodecSTT

Phone codec — 284 clips through telephony compression

stt-v3STT

Conversational clips — 897 spontaneous voice-agent turns from pipecat's stt-benchmark-data

tts-v1TTS

Text prompts — 30 short text inputs for synthesis

Clean studio audio is what most speech-to-text providers train on, so almost everyone scores well on it. We test that too, but we don't stop there. We pull open-source datasets that cover the conditions real calls actually run into: accents, background noise, reverb, mic distance, clipping, and phone-line compression, plus spontaneous conversational turns full of the fragments and fillers of real speech. Every clip is loudness-normalized to −20 dBFS RMS and locked to a SHA-256 hash, so the audio a provider hears stays identical from one run to the next.

Each scheduled run draws a random sample from its manifest and every model scores on that identical draw, so comparisons within a run are apples-to-apples while the full set is covered over time. Retired datasets stay published for historical reproducibility.

TTFATTS

Time to First Audio — first audio chunk arrival plus any leading silence before the first audible sample

TTFTSTT

Time to First Token — how quickly partial transcripts start streaming

TTFSSTT

Time to Final Segment — from a shared VAD end-of-speech anchor to the final transcript

WERSTT

Word Error Rate — scored after Whisper's EnglishTextNormalizer on both reference and hypothesis

WERTTS

Word Error Rate — synthesized audio transcribed by a fixed ASR model, scored against the input text

Latency metrics measure what a voice agent actually waits on. TTFA counts the delay a listener would notice, including any silence before the first audible sample, not just the moment the first bytes arrive. TTFS starts every provider from the same end-of-speech instant, picked by one shared voice-activity model, so no one gets an edge from where they decide the speaker stopped.

Text-to-speech gets a WER too: we transcribe each provider's synthesized audio with one fixed ASR model (OpenAI's whisper-1) and score it against the input text, so the number measures whether the speech is intelligible and complete. The transcriber's own mistakes put a small floor under every provider equally, and both directions share one scoring pipeline.

We exclude connection setup (TCP, TLS, and protocol handshakes) the same way for every provider, so the numbers reflect the model and not the network. Aggregates show the full spread from p25 to p99 next to the mean, and leaderboards rank by the median.

model_id

Exact model versions — every provider entry uses the versioned identifier the provider exposes, never an alias

norm_version

Versioned pipeline — a change to WER methodology never mixes with rows produced before it

runner_sha

Traceable runs — every run records the runner commit that produced it

Scoring runs on standard, lock-file-pinned libraries: jiwer for edit distance and whisper-normalizer for text normalization. When we change the methodology, we mark results from before and after as incomparable instead of quietly blending them.

The runner, the dataset manifests, and the methodology docs are all open source, and every audio file is checked against its SHA-256 hash on fetch. Every number on this site can be reproduced straight from the repository.