What is speech recognition (ASR) and how does it work?

You read your account number to your bank's phone line, clearly, the way you'd read it to a person. "Sorry, I didn't catch that." You say it again, slower. Same answer.

An hour later you dictate a text message while walking the dog, and your phone gets every word right, commas included.

Both moments run on the same technology: speech recognition. Why it nails one and fumbles the other comes down to the audio it hears and the words you need it to get right.

What is speech recognition?

Speech recognition is technology that turns spoken words into written text. You'll also see it called automatic speech recognition (ASR), speech-to-text (STT), or voice-to-text.

All four names describe the same thing: audio goes in, a transcript comes out.

In 1952, a circuit built at Bell Labs recognized spoken digits from a single speaker. Today's systems are expected to handle any word from any caller.

Speech recognition writes down what you said. It doesn't understand what you meant, which is a separate job with its own name.

Speech recognition vs voice recognition, text-to-speech, and keyword spotting

Speech recognition answers one question about a voice, "what was said?", and the terms people confuse it with each answer a different one.

A voice agent uses several of these at once. Speech recognition hears the caller, language understanding figures out the request, and text-to-speech speaks the reply.

How does speech recognition work?

Speech recognition works by turning sound into numbers, matching those numbers to speech sounds, and choosing the most likely words. Follow one sentence a caller might say: "I'd like to move my appointment to Friday at 3:15."

The audio is captured as numbers

A microphone or phone line measures the sound wave thousands of times a second. A standard phone call takes 8,000 samples a second, and each one is stored as a number.

The sound becomes a picture of frequencies

Speech recognition doesn't work on the raw wave. It slices the audio into pieces a few hundredths of a second long and turns them into a spectrogram, a picture of which frequencies are loud at each moment.

An acoustic model matches sounds to speech

An acoustic model learns which patterns in the spectrogram belong to which speech sounds. Trained on recordings of many different speakers, it can match the "fr" and "ai" of "Friday" across voices and accents.

A language model picks the likeliest words

Sounds alone leave ties. "To Friday" and "two Friday" sound identical, so a language model scores which word sequence people actually say and picks "to Friday."

The text gets cleaned up

The last step formats the transcript for people to read. It adds capitals and punctuation and writes numbers as digits, so "three fifteen" becomes "3:15."

Older systems vs end-to-end models

Older speech recognition ran these steps as separate models, each built and tuned on its own. Most modern systems are end-to-end: one neural network trained on audio paired with transcripts learns all the steps at once.

OpenAI trained its Whisper model on 680,000 hours of labeled audio, and it held up far better on accents and background noise than models trained on smaller, cleaner datasets.

End-to-end models can also fail in a new way. Because they generate text rather than look words up, they occasionally write words nobody said.

What are the types of speech recognition?

Speech recognition systems differ in when they transcribe, whose voice they expect, and how many languages they handle.

Real-time vs recorded

Real-time, or streaming, speech recognition transcribes while you talk and returns words a fraction of a second after you say them. Live captions and voice agents need it.

Recorded, or batch, recognition works on a finished file, such as a meeting recording. It can use the whole recording for context, which usually makes it a little more accurate.

Speaker-independent vs trained to one voice

Almost every system today is speaker-independent: it works for anyone, straight away. Older dictation software asked you to read passages aloud first so it could learn your voice.

Single-language vs multilingual

Many modern models handle dozens of languages, and some follow a speaker who switches languages mid-sentence. Accuracy still varies a lot from one language to the next.

A 2026 phone-call benchmark counted each time a caller spoke as a turn. It used an AI judge to flag transcripts with an error that changed what the caller meant.

On English turns, the systems tested made such errors 2.8% to 5.8% of the time. On Mandarin turns, the rate ran from 31.2% to 52.7%.

Speech recognition examples you already use

Speech recognition sits behind most features that let you talk to a device instead of typing:

• Voice typing turns speech into text messages, emails, and notes on phones and laptops.
• Live captions put subtitles on videos, video calls, and TV broadcasts.
• Voice assistants in phones, cars, and smart speakers transcribe your request before they act on it.
• Doctors and lawyers dictate notes and records instead of typing them.
• Phone menus and contact centers route callers by what they say and transcribe calls for records and quality review.
• AI voice agents answer and make phone calls, with speech recognition as their ears.

How accurate is speech recognition?

On clean recordings, a strong speech recognition model gets about 3 words in 100 wrong. In OpenAI's Whisper research, the same model got 2.7% of words wrong on audiobooks and 13.8% to 17.6% on recorded phone conversations.

Accuracy is usually measured as word error rate (WER): the share of words the system got wrong, counting words it swapped, dropped, or added. One wrong word in a 20-word sentence is a 5% WER.

In 2017, Microsoft reported a 5.1% WER on Switchboard, a standard set of recorded phone conversations. On that test set, Microsoft measured professional transcribers at 5.9%.

That system was trained on the same kind of recordings it was tested on. The higher figures on this page come from models tested on calls they weren't tuned for, which is closer to what a new set of calls will get.

A good score can still get the account number wrong

Word error rate counts every word the same. Getting "Friday" wrong costs as much as getting "the" wrong, so a transcript can score well and still miss the one word you needed.

The 2026 phone-call benchmark measured exactly that, on calls to an AI banking agent, averaged across six transcription services and all five languages it tested.

When callers gave a name, email address, code, or amount, those turns scored better on word error rate than the rest of the call: 15.0% vs 18.7%.

Those same turns were far more likely to contain an error that changed what the caller meant: 28.6% vs 17.0%.

The callers were people role-playing fictional customers, not real ones, and the study counted any misspelled name or code as a meaning error. Read the exact numbers loosely. The pattern is the point.

Why phone calls are hard to transcribe

Phone calls are harder to transcribe than clean recordings because a call carries less of your voice, compresses what's left, and often comes from a noisy place.

The Whisper gap above, roughly five to six times as many errors on calls as on audiobooks, has two parts. One is the phone line. The other is that people chat differently from how they read aloud. A real call has both.

A phone line keeps only part of your voice

Sound frequency is measured in hertz (Hz), vibrations per second.

Because a standard call samples sound 8,000 times a second, it can only carry frequencies up to 4,000 Hz, and the ITU-T G.712 standard sets the phone band at about 300 to 3,400 Hz.

The hiss of "s" peaks far higher, near 7,000 Hz in one study of English fricatives. The weaker hiss of "f" spreads across the same high range.

Much of what separates "F" from "S" never makes it down the line. That's why people spelling a name on a call say "S as in Sam."

Compression squeezes it further

Calls are compressed by codecs, the formats that shrink audio so it travels cheaply across the network. Some codecs drop detail a listener barely misses but a model relies on. Lost data packets can clip whole syllables.

Calls happen in noisy places

People call from cars, kitchens, and sidewalks, on speakerphone, with the TV on or someone else talking. Speech recognition has to pick one voice out of all of it.

The caller is harder to hear than the agent

On real customer support calls, speech recognition often does worse on the customer's side of the line.

In a 2026 study of support calls in Indian-accented English, a commercial telephony model got 29.7% of customer words wrong and 25.0% of agent words wrong.

The researchers' explanation is that agents sit in quiet offices with headsets, while customers call from noisy places and give shorter answers. That's their reading, not something the study measured.

Callers say the hardest things to transcribe

Callers need account numbers, spelled names, and addresses transcribed exactly. Those are the kind of words, such as names, codes, and amounts, that the banking benchmark found most likely to come out with the meaning changed.

Speech recognition in voice agents and chatbots

In an AI voice agent, speech recognition is the ears. It turns your words into text, a language model decides what to say, and text-to-speech says it out loud.

Chatbots with a microphone button use the same first step: speech-to-text turns what you say into the text the chatbot reads.

On a live call, speech recognition has two extra jobs: keeping up with you, and knowing when you've finished.

It has to keep up

People answer each other fast. A study of 10 languages found that the average gap before someone answers a yes-or-no question is about 200 milliseconds, a fifth of a second. The most common gap was no gap at all.

So speech recognition in a voice agent streams: it transcribes while you talk. By the time you stop, the agent already has most of your sentence.

It has to know when you've finished

Deciding when you've finished talking is called endpointing. Silence is a poor signal for it, because people pause mid-thought: "My account number is... hang on... 4417." A system that waits for silence can't tell thinking from done.

In a 2026 turn-taking benchmark, a detector that relied on silence alone decided the speaker had finished during 52.5% of pauses where the speaker was about to carry on.

A model trained to predict the end of a turn made that mistake 5.5% of the time, and took about 90 milliseconds longer to decide than the silence detector.

Wait too little and the agent interrupts you. Wait too long and every reply feels slow.

Where speech recognition still struggles

Speech recognition still struggles with unfamiliar accents, rare words, overlapping voices, mixed languages, and silence.

Accents and dialects

Accuracy varies by accent, dialect, and speaker group, because a model learns from the speech it was trained on. Accents that are rare in that training data tend to get more errors.

Names, numbers, and rare words

Surnames, product names, and medication names are words a model has rarely or never heard. The same step that turns "two Friday" into "to Friday" pushes an unusual surname toward a common word.

Account numbers fail differently. Any digit can follow any other, so the language model has no context to lean on, and one misheard digit breaks the whole number.

Many systems accept a custom vocabulary, a list of expected terms such as product names, which helps with words known in advance.

It doesn't help with account numbers. Reading the number back for the caller to confirm, or collecting it on the keypad, works better.

People talking over each other

When two people talk at once, speech recognition tends to drop words from one voice or blend both into one line. Recording each side of a call on its own channel reduces the problem.

Switching languages mid-sentence

Bilingual callers often switch languages mid-sentence, while many models expect one language per call. Models built for code-switching, the move between languages, handle it better.

Writing words nobody said

End-to-end models sometimes invent text, which researchers call hallucination.

A 2024 study ran recordings of people with aphasia, a language disorder, and of other speakers through Whisper in 2023. About 1 in 70 transcripts (1.4%) contained a phrase nobody said, more often when speakers paused for long stretches.

Of those invented phrases, 38% were harmful, such as made-up violence, invented personal details, or false claims of authority.

When the researchers re-ran the same clips in December 2023, after an update to Whisper, most of the hallucinations were gone.

Audio with no speech still trips up newer models. Fed recordings of noise, music, and other sounds, Whisper's large-v3 model still wrote text 40% of the time in a 2025 test, most often "thank you."

The common fix is to filter out stretches with no speech before transcribing.

Running speech recognition on live calls

Running speech recognition on live calls takes streaming transcription, audio that reaches the model quickly and intact, and a way to test models on real calls.

Streaming keeps the agent close behind the caller. Every network a call crosses on its way to the model adds delay and another chance to lose packets. And only recordings of real calls show how a model handles your callers.

Telnyx's STT Router, a transcription API that connects to several speech recognition engines, transcribes calls in the same facility that handles the call.

Telnyx carries calls on its own private network, so a call's audio doesn't take an extra trip across the public internet to a separate transcription service.

If you're building on the Telnyx speech-to-text API, you can compare engines on your own calls and switch with a configuration change instead of a rebuild. Rates are on the speech-to-text pricing page.

Frequently asked questions about speech recognition

If you're choosing an engine, the accuracy number that counts is the one from your own calls. Test the Telnyx speech-to-text API on your recordings and compare engines on the words that matter to you.