The Frankenstack problem: why your voice AI breaks in production

Your voice AI pipeline has 800 milliseconds of latency, and most of it never touches a model.

That number is not hypothetical. Retell, a well-run orchestration platform, publishes response times "as low as 800ms" as a selling point. Twilio publishes half a second at the median for its speech relay, and that figure excludes the LLM entirely. When the best published numbers in the category are between half a second and a full second for a partial pipeline, the problem is not the models. The problem is the architecture underneath them.

The architecture most teams actually run

Most production voice agents today run on what we call the Frankenstack: four to six vendors assembled into a single pipeline. A carrier or CPaaS reseller provides the phone number and SIP trunk. An orchestration platform glues the pieces together. A speech-to-text vendor transcribes the caller. An LLM provider, usually in a distant cloud region, generates the response. A text-to-speech vendor synthesizes the reply. Sometimes a separate media layer moves the audio between all of them.

Each of those companies is good at its layer. That is exactly why teams pick them. The failure is not in any component. It is in the seams.

What the seams cost you

Every vendor boundary is a network hop across the public internet, and each hop adds 30 to 80 milliseconds before any model does any work. Chain four or five of them and you have added 150 to 400 milliseconds of pure network overhead per conversational turn. Speed of light in fiber is roughly 5 milliseconds per thousand kilometers, so if your LLM runs in a cloud region far from where the call terminates, physics is against you before the first token generates. You cannot prompt-engineer your way out of a round trip.

Reliability compounds the same way. Five vendors each running at 99.9% uptime yield 99.5% compound availability, which is about four and a half hours of downtime per month, and any one vendor's incident is your incident. When something does break at 2am, the debugging is worse than the outage: five dashboards, five support queues, no end-to-end trace, and every vendor's telemetry ends at its own boundary. The telephony provider blames the STT vendor, the STT vendor points at the LLM, and the customer becomes the debugger.

What voice AI actually costs per minute

Each vendor in the chain takes its margin, and four vendors deep you are paying four margin layers on every call. The per-component numbers make it concrete: Telnyx text to speech starts at $0.000003 per character while ElevenLabs starts at $0.00003, ten times more for the same task, because one runs on owned infrastructure and the other resells compute. SIP trunking shows the same pattern at $0.0032 per minute on Telnyx against $0.013 on Twilio.

An integrated stack prices differently because there is nothing to stack. $0.05 per minute covers orchestration, call control, speech to text, and Telnyx text to speech, with hosted models billed per token from $0.60 per million input tokens rather than marked up per minute, and the full per-component rates are on the voice AI pricing page. Stack enough margin layers and typical teams save around 45% moving to an integrated platform, not because anyone is discounting, but because the middlemen are gone.

And there is a cost no invoice shows: the engineering time to integrate four APIs, re-integrate them every time one changes, and carry the operational load of a distributed system you did not set out to build.

The alternative is architectural, not incremental

Telnyx runs the entire voice AI pipeline on infrastructure we own. Calls enter on our licensed carrier network. Speech-to-text, agent orchestration, and text-to-speech run on the same platform. LLM inference runs on Telnyx-owned GPUs deployed at the network edge, in the same operational domain as the telephony switch rather than a cloud region several networks away.

Audio that enters the network is transcribed, routed to a model, synthesized, and returned without ever crossing to another provider. Zero inter-provider hops is not an optimization, it is a property of the architecture, and it is the property a Frankenstack structurally cannot have.

Ownership also collapses the operational story. One SLA covers the call path end to end. One trace runs from PSTN ingress through inference to the synthesized reply. When something breaks, one team owns the whole path, which is the difference between a fix and a week of vendor ping-pong.

Proof, not promises

Turn-taking on the integrated pipeline stays inside conversational rhythm, with end-to-end round-trip latency under 500 milliseconds measured across the full loop, against the 500 to 800 milliseconds the leading multi-vendor platforms publish for partial pipelines that exclude the LLM leg or the telephony hop. Independent verification matters more than our own numbers, which is why we contributed the ExternalAgentProvider interface to ServiceNow's EVA benchmark so hosted voice agents, including ours, can be measured end to end by a third-party framework rather than self-reported demos. Compound the availability math, the per-component pricing above, and the published latency figures of the multi-vendor platforms, and the pattern holds: every claim in this post traces to a pricing page, a published benchmark, or arithmetic you can redo yourself.

What this looks like when you build

A production agent on Telnyx is one integration. You provision a number, define the assistant, and the platform handles speech, inference, and voice on one API with one bill. Warm transfer, multi-participant calls, and conversation insights are platform features rather than integration projects, and the same infrastructure carries your SMS confirmations and follow-ups so the agent's whole operation stays in one system. Compliance behaves the same way: SOC 2 Type II, HIPAA, PCI, ISO, and GDPR are properties of the platform, not add-ons priced per vendor.

The Frankenstack was a reasonable way to prototype voice AI in 2024. It is a liability in production in 2026, and the teams shipping the fastest agents have already stopped stitching. If you want to see the difference in your own stack, run the numbers on your current pipeline: add up the per-minute meters, count the network hops, and ask who you would call at 2am. Then build the same agent on telnyx.com and compare.