Real-time voice inference: where every millisecond goes on a live call

What is real-time voice inference?

A caller starts talking, and their voice leaves the handset in small audio frames, one every 20ms, while the sentence is still forming. Real-time voice inference means each model in your pipeline starts work on that partial input and streams its output onward before the caller has finished.

Should my voice agent be cascaded or speech-to-speech?

• Callers read out account numbers: run a cascaded pipeline. The text between models shows exactly which digits were caught, so a misheard number is traceable.
• Want to replace one model later: keep speech-to-text, LLM and text-to-speech as separate models. Replacing one leaves the other two untouched, for the price of two handoffs.
• Handoffs eat most of your measured wait: try a speech-to-speech model. Audio in and audio out drops both text handoffs, but you give up the transcript for debugging.
• Cascaded, models spread across clouds: run all three on Telnyx Inference. Speech-to-text, the LLM and text-to-speech share one network, so cross-cloud round trips leave the budget.

Why request-response inference breaks a live call

Request-response inference waits for the whole recording. A batch transcription API takes a finished file and returns a finished transcript, so every second of processing reaches the caller as dead air. Text chat can absorb that delay because the user is reading. A phone call can't, which is the main split between conversational vs voice AI.

Cascaded pipeline or speech-to-speech model?

Real-time pipelines come in two shapes, and the budget in the next section applies to both:

• Cascaded (speech-to-text, LLM, text-to-speech): text passes between three models. You can swap any model and read the transcript, but you pay two handoffs.
• Speech-to-speech model: audio goes in and audio comes out. Two handoffs disappear, and so does your view of what the model heard, which makes a misheard account number harder to debug.

Either way, a phone call adds something a browser demo never shows: before any model sees the caller's audio, it has crossed a carrier network. On Telnyx, the language model runs as OpenAI-compatible chat completions on open-weight models hosted on GPUs Telnyx owns, and speech-to-text and text-to-speech run on the same platform.

Run your whole voice pipeline on one network

Telnyx runs OpenAI-compatible inference, speech-to-text, and text-to-speech on the same platform, so every model in your pipeline shares one network.

Try Telnyx Inference

Where the latency budget goes in a real-time voice inference pipeline

Your caller's wait runs from the moment they stop talking to the first syllable of the reply. Six hops sit inside that window, and two of them belong to the phone network.

The six hops between the caller stopping and the reply starting

Walk the clock forward from the caller's last word and score your own agent against each row:

Six hops in one phone turn

Why the network is the hop nobody budgets

Teams fighting dead air usually start with the model. They swap in a faster one, rerun the benchmark, and watch it improve while live calls still feel slow, because the benchmark only covered rows three through five.

Published vendor latency budgets start at the microphone and leave out both carrier legs. ITU-T G.114 recommends keeping one-way mouth-to-ear delay to 150ms for most applications, and every phone turn pays one-way delay twice: once on the way in and once on the way out. Those legs can use up a large share of a natural-feeling turn before a single model runs, which is why end-to-end measurement belongs among your voice AI production principles.

The telephony hops before voice inference even starts

Before any model hears a phone caller, a carrier has routed, packetized, and terminated their audio. That path decides how much budget is left and how clear the audio is when it arrives.

From PSTN or SIP to the media plane

A call travels over the public switched telephone network (PSTN) or a SIP trunk to the carrier, whose media plane terminates the Real-time Transport Protocol (RTP) stream. Only then can the audio reach a speech-to-text model, and the reply makes the same trip in reverse. WebRTC clients skip the phone network, but a server still has to terminate their media.

If your telephony provider, speech-to-text vendor, and language model sit with three different companies, the audio crosses the public internet between each of them, adding distance and jitter every time. Telnyx runs text-to-speech, voice AI, and telephony on the same infrastructure, with one API key and one bill. Calls arrive through the Voice API on a carrier-grade network Telnyx owns, so the carrier leg and the inference leg share one path.

8 kHz G.711 audio and what your STT model expects

Most phone audio arrives as G.711, sampled at 8 kHz in 20ms frames. Many speech-to-text models expect 16 kHz PCM, and upsampling can't restore the frequencies the narrowband line already discarded. Accuracy on names and numbers is partly settled before inference begins, so when you compare engines from a voice to text software roundup, test them on recorded 8 kHz phone audio.

Where should real-time voice inference run?

Judge every hosting option by one question: how many network crossings sit between your caller's carrier and the GPU running your model?

Edge PoP, hyperscaler GPU region, or colocated with the network

• Edge point of presence (PoP): close to the caller, with one crossing from the carrier, but limited GPU capacity for large models. Edge inference explained covers when it fits.
• Hyperscaler GPU region: plenty of GPUs, with two or more crossings: telephony provider to cloud, and often cloud to a TTS vendor.
• GPUs colocated with the carrier network: the media plane and the model sit on the same network, with no crossing between them. The trade is a curated model library.

A fallback provider in another cloud follows the same arithmetic: it adds a crossing, so it can raise turn latency even when its model answers faster. The penalty applies one component at a time, too.

What colocating GPUs with the carrier network removes

Telnyx runs models on GPUs colocated with its network, in-region across the Americas, Europe, MENA, and APAC. No inter-provider hops sit between the media plane and the GPU. Capacity autoscales with no cold starts, so a traffic spike doesn't leave callers waiting for a GPU to warm up. If your agent runs on LiveKit, the LiveKit integration docs show how to point it at Telnyx.

Which hop should you cut first?

• STT, LLM, and TTS on three clouds: consolidate onto one network before tuning any model.
• Telephony with one provider, GPUs with another: move inference onto the telephony network.
• One US region, callers in Europe or APAC: run inference in the caller's region.
• One provider, one region, still talking over callers: look at endpointing and jitter.

Keeping LLM latency out of real-time voice inference

Your caller hears nothing until the first byte of synthesized audio, and that byte waits on the model's first sentence. The number to optimize is time to first token, and the technique is streaming.

Stream tokens into TTS before the sentence is finished

Send the first complete sentence to TTS while generation continues. Waiting for the full reply adds the entire generation time to the caller's wait. Context grows too: a ten-minute call carries a much longer prompt than its first turn, so summarize or trim history between turns.

Pick a model and a service tier for a live call

A smaller model on fast capacity usually beats a larger model that queues. Telnyx hosts a curated inference model library, and each request can set a service_tier, with rates on the inference pricing page:

• Default: standard capacity at standard rates, used when you omit the field.
• Priority: capacity optimized for low-latency interactive work, including voice, with the same model and context window. Live on select models.
• Flex: the lowest rates, for background jobs that can wait minutes.

Because the API is OpenAI-compatible, moving an agent from another provider means changing the base URL. Per-tier latency figures aren't published, so compare Priority against Default on your own calls.

Example: Telnyx inference inside a live translated call

This translation bridge sample connects two callers who speak different languages, using Voice Call Control and AI Inference. The transcript enters the model in translate(), and the call.gather.ended handler speaks the output to the other caller:

The sample uses a two-second silence timeout, which suits translation but is far too slow for an agent. That is where sentence streaming and a tighter endpoint come in.

Barge-in and turn detection in a real-time voice pipeline

The caller pauses to think, the agent starts answering, the caller resumes, and now both are talking. That overlap is a timing problem, and a faster model rarely fixes it.

Why your agent talks over callers

Turn detection decided the pause ended the turn. A short silence threshold makes that mistake often, and a long one trades talk-over for dead air. Jitter on the carrier leg stretches and compresses the very gaps the detector measures, so the same threshold behaves differently from call to call. The start speaking plan guide covers the timing controls.

What has to happen when a caller interrupts

1. Stop TTS playback.
2. Cancel the in-flight LLM request.
3. Discard the agent's own audio from the STT input, so echo isn't transcribed as caller speech.
4. Record how much of the reply the caller heard, and update the context to match.

Keyword matching on wait or actually misses most interruptions, and skipping step four leaves the model believing the caller heard a reply they never heard.

Compliance and data residency for real-time voice inference

Before a healthcare or finance agent goes live, compliance will ask where the audio terminates, where the model runs, and what is kept. Telnyx runs inference in-region across the Americas, Europe, MENA, and APAC, with LATAM coming soon, and there is no cross-border routing. Requests run with zero data retention and no training on your data.

The voice agents for healthcare guide and the healthcare voice AI page cover patient calls. Recording consent rules vary by jurisdiction, and the Australia data sovereignty breakdown walks through one market.

How to measure real-time voice inference on live calls

The clock starts when the caller stops speaking and stops when the first reply audio reaches them. Only a measurement on the call path captures both carrier legs, so time to first token is one input.

Metrics that predict hang-ups

Report turn latency at p50 and p95, since an average hides the slow turns callers remember:

• Turn latency: caller end of speech to first reply audio at the caller.
• Talk-over rate: turns where the agent started while the caller was still speaking.
• Dead-air rate: turns where the gap exceeded your threshold.
• First-turn hang-up rate: callers who left before or during the first reply.

Run these alongside your production checklist, and watch hang-ups on calls that route to a person, where the AI to human handoff adds a wait of its own.

Example: live transcription and sentiment from a call

This dashboard sample turns a Telnyx call into a stream of transcription events. Stamp each with its arrival time, and you can line up caller speech against when your agent starts playing audio:

Subtract logged model time from measured turn latency, and what remains is network and handoff time. Across a fleet, that split shows whether a slow week came from the network or the model, which matters most for high-volume contact center solutions.

Real-time voice inference FAQ

Why does my voice agent talk over callers, and is it latency or turn detection?

Usually turn detection. The agent starts speaking when its endpointing decides a pause ended the caller's turn, and jitter on the carrier leg makes those pauses look longer or shorter than they were. Latency makes it worse, since a slow reply lands after the caller has resumed. Tune the silence threshold and handle barge-in before you swap models.

Can I use a second inference provider as a fallback for my voice agent without rewriting it?

Yes, if both providers expose an OpenAI-compatible API. Telnyx inference is OpenAI-compatible, so pointing an agent at it means changing the base URL. Check where the fallback runs, though. A provider in another cloud adds a network crossing to every turn it serves, so it can raise latency even when its model is faster.

What latency makes a voice conversation feel natural?

There is no single agreed number, and published vendor thresholds disagree. ITU-T G.114 recommends keeping one-way mouth-to-ear delay to 150ms for most applications, and a phone turn pays one-way delay in each direction plus model time. Set your target as turn latency at p95 measured on live calls, and track talk-over and dead air alongside it.

Why can't standard request-response inference do real-time transcription?

Request-response inference needs the complete recording before it starts, so a transcript only exists after the caller stops talking. Real-time transcription streams audio frames into the model as they arrive and returns partial and final transcripts during speech. On a call, waiting for the whole recording turns every pause into dead air.

Should I use WebSockets or WebRTC for voice AI?

Use each where it fits. WebRTC suits browser and mobile clients, since it carries media over UDP with jitter buffering and echo cancellation built in. WebSockets suit server-to-server audio streaming, such as forwarding call audio to a model. For phone callers, the leg that reaches you is the carrier's RTP stream, so this choice only affects the hops after the media plane.