Why your voice AI agent is slow on phone calls, and how to fix it

Updated October 2026

Quick answer: A voice AI agent feels slow on a phone call when one stage of its turn runs long: the wait to decide the caller has finished, the model's first token, a tool call, speech synthesis, or the network path. Median voice AI turns take 1.3 to 1.7 seconds, against about 200 ms between human speakers. The fix starts with timing each stage across a few hundred real turns and addressing the slowest parts first.

Why voice AI agents feel slow on phone calls

Voice AI agents feel slow on phone calls because a typical turn takes six to eight times longer than the pause people leave for each other. Across ten languages, researchers measured an average gap of 208 ms between human speakers.

Compare this to phone calls with an AI agent on the other line: the average voice AI latency runs up to 1,740 ms in an independent test from Openbenchmarks.

Past roughly two seconds of silence a caller assumes the line dropped and starts talking over the agent.

In a listening study of answers between people, even a pause of more than about 600 ms before a "yes" made the answer sound reluctant. In a study of in-car voice assistants, users read an unnaturally late reply as a system error.

The 8 causes of voice AI delay, and how to fix each

The eight causes of voice AI delay are the end-of-turn wait, a one-stage-at-a-time pipeline, model inference, tool calls, network distance, cold starts, peak load and jitter. Each leaves a different pattern in a trace.

1. The end-of-turn wait

The end-of-turn wait is how long the system waits after the caller goes quiet before it treats the turn as finished. Engineers also call it endpointing or turn detection.

Too short a wait cuts callers off mid-thought. Too long a wait adds dead air to every turn. A caller reading out a phone number pauses between digits, while a caller answering yes or no does not.

On a Telnyx AI Assistant, the start-speaking plan holds these waits. Its documented defaults are 0.1 seconds after punctuation, 1.5 seconds without it and 0.5 seconds after a number.

Turn-taking speech models make their own prediction of when the caller has finished. With deepgram/flux on Telnyx, transcription settings such as eot_threshold and eot_timeout_ms control that prediction.

The start-speaking plan mainly applies to models without built-in turn detection, though Telnyx's docs differ on how fully. With Flux, Telnyx suggests keeping start-speaking delays low, around 0.1 seconds.

2. A pipeline that runs one stage at a time

A pipeline that finishes each stage before starting the next adds every stage's full time. Streaming lets the stages overlap: transcription runs while the caller talks, and speech synthesis starts on the first words of the reply.

In published research, the DDTSR framework, which starts speaking before transcription and reasoning finish, cut response latency by 19% to 51% against sequential pipelines.

Eager end-of-turn, on by default for Deepgram Flux on Telnyx, starts the model's reply before the speech model has confirmed the caller is done. Telnyx reports about 150 ms saved at the median when eager turns trigger.

On the slowest 5% of turns, the saving reached about 350 ms. If the transcript changes mid-turn, the assistant falls back to standard turn handling.

3. Model inference and a growing prompt

Model inference is the wait for the language model to start its reply. Inference latency on a voice call is usually measured as time to first token: the delay before the first word piece arrives.

On the Artificial Analysis leaderboard, the median for models with reasoning on is nearly three times the median for models without it.

The prompt adds to that wait, because the model reads the system prompt and the conversation so far before it answers. On a long call, a growing prompt can make late turns slower than early ones.

The fixes are a smaller or non-reasoning model for routine turns, a lean system prompt, and a summary of earlier turns in place of the full history.

4. Tool calls and backend lookups

A tool call leaves the caller in silence until the backend answers, because the agent waits for the result before it speaks.

OpenAI's Realtime prompting guide calls speaking while a tool runs "masking latency": the agent says one short line, such as "I'm checking that now," and calls the tool in the same moment.

In a 2025 study of LLM-powered virtual agents, natural fillers improved perceived response time, most of all at the longer delays tested, 4 and 6.5 seconds.

The authors of a 2026 study on dialogue robots note the limit: fixed fillers become unnatural over time.

On a Telnyx AI Assistant, filler messages play scripted phrases during synchronous webhook and MCP (Model Context Protocol) tool calls. One plays as soon as the webhook call begins. More play after delays set between 100 ms and 120 seconds.

For backends that consistently take longer, async tools return at once, so the conversation keeps going while the work finishes.

Caching frequent lookups removes the wait for repeat questions. When the reply needs the result, such as an open appointment slot, the wait only shrinks at the source: a faster backend, or a lookup started earlier in the call.

5. Network distance and cross-vendor telephony

Every leg between the caller, the phone network and each AI service adds a round trip. A stack spread across several vendors adds more legs, because the audio leaves one network and enters another at each handoff.

On Azure's backbone, median round trips run about 70 ms across the US and 71 to 118 ms from the eastern and south-central US to Western Europe.

Legacy phone systems add delay on top of distance, before the AI hears anything. Each codec conversion adds its own processing delay.

“You can improve each component and still end up with a slow-feeling call if the full loop is spread across too many places. In voice AI, the user experiences the system end to end.” James Whedbee, VP of Engineering, Telnyx

The fix is fewer boundaries: inference in the caller's region, and telephony and AI on one network where possible. A stitched-together stack, with telephony at one vendor and speech and models at others, adds a hop at each seam.

Running call control on the Telnyx Voice API alongside Telnyx's own speech and inference removes those seams.

6. Cold starts

A cold start is the delay when a request reaches a model that has scaled down to zero and has to load before it can answer.

Hugging Face scales an idle endpoint to zero after 15 minutes without requests. The symptom is distinctive: the first call after a quiet period is slow, while later calls are fast.

The fix is to keep models warm for live agents: no scale-to-zero, a minimum of running capacity, and a warm-up before known peaks such as the start of business hours.

7. Concurrency and peak load

Peak load slows an agent when calls compete for the same compute, network capacity and third-party rate limits, so work waits in a queue. A single test call cannot show it, because queueing only appears when many calls run at once.

The same pressure can surface as dropped calls and choppy audio. The fix is a load test above the expected peak, a check of every dependency's rate limits, and capacity added before the slowest turns start to climb.

8. Jitter, packet loss and the media path

Jitter and packet loss make an agent sound slow and garbled at once, because the system buffers late audio and patches lost audio before the caller hears it. Jitter is variation in when audio packets arrive.

A jitter buffer, which holds incoming audio briefly so uneven packets play out evenly, adds about half its depth on average: 25 ms for a 50 ms buffer, by ITU-T planning values. A deeper buffer smooths more and adds more delay.

The fixes are an efficient codec such as Opus, a tuned jitter buffer, packet-loss concealment and a private media path instead of the public internet.

For SIP trunks connected to Telnyx, an adaptive jitter buffer can be enabled per connection through the API, with a default depth range of 60 to 200 ms.

Match the symptom to the cause

The pattern of the slowness points to the stage to check first.

Symptoms and the stage to check first

How to measure and fix voice AI latency

Measuring voice AI latency means timing each stage of a few hundred real turns and judging each stage on its 95th percentile, or p95: the time one turn in twenty meets or exceeds.

Trace one turn

Tracing one turn means logging when each stage starts and ends, then subtracting the timestamps. For the engineer running the agent, the trace takes five steps:

1. Log five timestamps per turn from platform logs: speech end, final transcript, turn complete, the first token of the reply, and first synthesized audio, plus tool start and end when a tool runs.
2. Record test calls from a separate line. Time each turn on that recording, from the end of the caller's speech to the first audio of the reply, the way Openbenchmarks does.
3. Subtract the logged timestamps to get each stage. The caller-side total minus their sum is the time on the network legs. No clock alignment is needed, because each total comes from one clock.
4. Compare each stage with a budget set from the agent's own baseline, using the stage table's ranges as reference points. Stage p95s do not add up to the turn's p95, so judge the total on its own.
5. Fix the largest stage, redeploy, and trace again.

Example: tracing one slow turn

In this illustrative trace, a slow turn on default settings takes 2,920 ms. Two changes bring it to 1,620 ms.

Illustrative trace: one slow turn before and after two changes

The end-of-turn wait was the largest stage. The caller's last sentence produced a transcript with no punctuation, so the default 1.5-second wait ran before the turn could complete.

Lowering on_no_punctuation_seconds to 0.4 seconds saves 1,100 ms on turns like this one.

Turns that end with punctuation already wait only 0.1 seconds, so the overall saving depends on the share of turns that end without it, which is worth measuring first.

The block applies to speech models without built-in turn detection. With deepgram/flux, the transcription settings described under the end-of-turn wait apply instead. Every other value stays at its documented default:

The lower wait needs testing on real calls. on_number_seconds stays at 0.5 so callers reading out digits are not cut off.

Check whether latency costs calls

The check compares turns: of the turns slower than the agent's p95, what share were followed by a hang-up or a talk-over, against the same share for faster turns.

We found no published study that puts a number on latency-driven hang-ups, so the agent's own logs are the evidence.

Talk-over rate belongs next to containment, the share of calls the agent resolves without a person, in any voice AI evaluation. Long pauses drive both talk-overs and hang-ups, per Openbenchmarks.

Where the time goes in each turn

A voice AI turn is five consecutive pauses: the final transcript, the end-of-turn wait, the first token, the first synthesized audio and the network legs. On default settings, the end-of-turn wait can be the largest of the five.

Tool calls add an extra step.

Reference ranges for each stage of a voice AI turn

The ranges come from different sources and statistics, so they work as reference points, not as a budget.

With timer-based endpointing on Telnyx, the end-of-turn wait starts once the transcript ends, so it adds to transcription time rather than overlapping it.

Voice AI platforms with low latency

Telnyx had the lowest median of the five voice AI platforms in Openbenchmarks' caller-side test, 1,296 ms, with a p95 of 1,856 ms. None of the five answered in under a second at the median.

ElevenLabs had the lowest p95, 1,768 ms, with a median of 1,424 ms, the smallest gap between the two.

Openbenchmarks frames the choice by call type: the median for ordinary back-and-forth, the p95 when a caller must never think the line dropped.

The voice AI latency benchmarks page compares all five platforms and the method behind each test.

Alex Cohen CEO, Hello Patient

Customer story: Hello Patient Hello Patient uses Telnyx Voice AI infrastructure to power AI agents across voice, text, and web chat for healthcare workflows. “We're seeing more appointments being booked with fewer staff needed to run call centers and traditional phone lines.” More than 5 million patient conversations powered in under two years. Read customer story

How Telnyx shortens the path

Telnyx cuts the boundary count by running the carrier network and the AI together. Telnyx is a licensed carrier with its own private global backbone.

In its inference regions, the Telnyx Voice AI agent platform co-locates GPU inference with telephony points of presence. Those are the sites where calls enter the Telnyx network.

A turn can then run as one end-to-end voice AI stack, crossing fewer vendor boundaries where delay and jitter build up.

That shorter path is what AI agent infrastructure means in practice: Telnyx Voice AI agents run on the network that carries the call.

In its co-located infrastructure write-up, Telnyx publishes a figure of less than one second from the moment a caller finishes speaking until the reply is heard.

Frequently asked questions

Why does my voice AI agent have such high latency during phone calls?

A voice AI agent has high latency on phone calls when one stage of its turn runs over: the end-of-turn wait, the model's first token, a growing prompt, a slow tool call, the network legs between vendors, or queueing at peak load.

Tracing a few hundred turns stage by stage shows which one. On default Telnyx settings, the end-of-turn wait alone can reach 1.5 seconds.

How do I reduce latency in voice agents?

Reducing voice agent latency starts with measurement: timing each stage of real turns and fixing the slowest first.

The usual fixes are a shorter end-of-turn wait, overlapping stages, a smaller model or a leaner prompt, spoken fillers during slow tool calls, warm models, and inference in the caller's region on one network.

How do I fix cold start latency for AI agents?

Cold start latency goes away when the model stays loaded. Live voice agents need hosting that does not scale to zero, a minimum of running capacity and a warm-up before known peaks.

On scale-to-zero platforms, the first request after an idle period waits for the model to load, which providers describe as seconds to minutes.

Why does my AI voice agent drop calls or have poor audio quality?

An AI voice agent drops calls or loses audio quality at scale when calls compete for shared compute, network capacity and rate limits.

Requests start to queue. The same pressure shows up as jitter, packet loss and dropped calls. A load test above the expected peak finds the limit before callers do.

What causes audio lag in real-time voice AI?

Audio lag in real-time voice AI comes from audio packets arriving late, out of order or not at all. The system buffers and patches the gaps, which callers hear as delay and distortion together.

The first things to check are jitter and packet-loss stats for the call, the codec in use and the jitter buffer setting.

How do I minimize the pause when my voice agent runs a tool mid-call?

The pause during a lookup shrinks when the agent speaks while the tool runs. A short line such as "Let me check that" covers the wait. Research on conversational agents shows that natural fillers improve perceived response time.

On Telnyx, filler messages play during webhook tool calls. Async tools keep the conversation going during longer lookups.