TTS architecture decides what breaks first

Most teams choose text-to-speech by listening to samples.

That is useful, but it is not enough. A voice that sounds good in a demo can still be wrong for a live product. It may stream poorly. It may clone well but add too much latency. It may sound natural in English and fall apart across languages. It may be cheap for short prompts and expensive once every customer interaction turns into generated audio.

The reason is simple: text-to-speech quality is not only a voice-design question. It is an architecture question.

The architecture underneath the voice decides what the model is good at, what it struggles with, and what will break first in production. Some systems treat TTS as spectrogram prediction. Some treat it as audio-token generation. Some treat it as denoising. Some use recurrent state instead of attention. A newer category puts speech inside a larger foundation model and makes voice behavior promptable.

Those choices matter more than the marketing page suggests. If you are building a real-time voice agent, latency and streaming stability matter more than a perfect studio sample. If you are building an audiobook product, long-form consistency and cloning quality matter more. If you are building internal narration, cost may dominate everything else.

This is the practical way to think about modern TTS architectures: not as a research taxonomy, but as a map of tradeoffs.

What is a TTS architecture?

A TTS architecture is the structural design of a text-to-speech system: how it converts text into audio, what intermediate representations it uses, and how the model is trained and served. It is not the voice, the language coverage, or the sample quality on a demo page. It is the engineering decision underneath all of those things.

Two TTS models can sound similar in a short clip and behave completely differently in production. One may stream audio continuously from the first token. Another may need to generate a full chunk before releasing anything. One may clone a voice from 3 seconds of audio. Another may need hours of training data per speaker. Those differences come from the architecture, not the voice library.

The old baseline was a two-stage pipeline

For years, most neural TTS systems followed the same basic pattern: one model predicted a mel spectrogram from text, then a vocoder converted that spectrogram into waveform audio.

Tacotron 2, FastSpeech 2, Glow-TTS, WaveNet, WaveGlow, and HiFi-GAN all belong in this world. The exact model changed, but the shape stayed familiar. First, an acoustic model turned text or phonemes into a time-frequency representation of speech. Then a vocoder generated the audio that people actually heard.

That split made sense. It separated the problem into pieces engineers could train, inspect, and improve independently. The acoustic model learned pronunciation, timing, and rough prosody. The vocoder learned how to turn that intermediate representation into natural-sounding audio.

The training path was also clean. The acoustic model trained on paired text and audio, usually by predicting spectrograms and durations. The vocoder trained on spectrogram and waveform pairs, often with adversarial losses and mel losses to make the output sound less flat.

The tradeoff is that two stages mean two failure points. Bad alignment in the acoustic model carries into the vocoder. A lossy spectrogram limits what the vocoder can recover. Latency also stacks: the system has to produce the intermediate representation before it can produce the waveform.

This architecture is still useful when you want something predictable, well understood, and cheap enough to serve at scale. It is less compelling when you need zero-shot voice cloning, richer speaker similarity, or the lowest possible live-call latency.

The old pipeline did not disappear because it was bad. It got boxed into the jobs where its tradeoffs are acceptable.

VAE and normalizing-flow models collapsed the pipeline

The next important move was to stop treating the acoustic model and vocoder as separate products.

VITS, VITS2, and NaturalSpeech 1 moved TTS toward end-to-end generation. Instead of training one model to predict a spectrogram and another to synthesize audio, the system learned a latent speech representation and generated audio more directly.

VITS is the clean example. It uses a conditional variational autoencoder. A text encoder builds a distribution over possible speech representations. A posterior encoder sees the real audio during training. A normalizing flow makes the latent space more expressive. A stochastic duration predictor handles the fact that the same sentence can be spoken in many valid ways. A decoder produces waveform audio.

That design matters because speech is not deterministic. The same text can be fast, slow, flat, emotional, clipped, careful, or conversational. A model that only learns a single average answer tends to sound averaged. The stochastic parts of VITS helped model that one-to-many mapping more naturally.

The upside is a single-stage system with fast parallel generation and no obvious spectrogram handoff between two separately trained models. For open-source deployments, VITS-family models remain practical because they are relatively efficient and well understood.

The downside is training stability. The reconstruction loss, KL loss, adversarial loss, duration behavior, and flow all have to balance. If that balance is off, the model can sound smeared or inconsistent. Voice cloning is also not native in the way people now expect from modern commercial systems. You usually need a separate speaker embedding path.

This camp is important because it pushed TTS past the old two-model pipeline. It is not where most of the current voice-cloning excitement lives.

Codec language models made voice cloning feel native

Neural codec language models changed the center of gravity.

Instead of predicting a continuous spectrogram or waveform, they convert audio into discrete tokens and then train a language model to predict those tokens. Once speech becomes tokens, TTS starts to look more like next-token prediction.

VALL-E made this idea visible. The system used a short voice prompt and generated speech in that speaker's voice. The trick was not magic cloning. It was a codec plus a language model.

The codec comes first. Systems such as EnCodec and SoundStream compress audio into discrete codebook tokens using residual vector quantization. The first codebook captures coarse information such as phonetic content and speaker identity. Later codebooks add finer acoustic detail. With enough codebooks, the token stream preserves much of what matters in the original audio at a lower bitrate.

Then the TTS model learns to predict those audio tokens from text and a speaker prompt. At inference time, a short sample of a target speaker gives the model enough context to generate new speech in a similar voice.

That is why this family became so important for AI voice models. It maps well to the way modern AI infrastructure already works. You can train large sequence models. You can scale with data and compute. You can use in-context learning behavior for voice prompts. You can build multilingual systems without inventing a different architecture for every language.

The cost is latency and error accumulation. Autoregressive systems generate tokens step by step. Longer output means more steps. More codebooks can mean better fidelity, but they also mean more tokens to predict. If the model drifts, mistakes can compound over a long generation.

This is the family to care about when speaker similarity, cloning from a short prompt, and multilingual quality are the main job. Many modern cloning systems appear to follow this broad pattern, although vendors do not always disclose the exact design.

For live agents, the question is sharper: can the provider stream the output fast enough, consistently enough, and cheaply enough once the model is inside a real call path?

Diffusion and flow matching optimize for controllable generation

Diffusion and flow-matching models come from a different instinct.

Instead of generating speech token by token, they start from noise or a simple prior and learn how to move toward clean speech. NaturalSpeech 2 and 3, Voicebox, E2-TTS, F5-TTS, and Matcha-TTS sit in this camp.

The practical difference is that generation is not autoregressive in the same way. The model can work over a chunk of speech in parallel and refine it over a fixed number of steps. In diffusion, the model learns to reverse a noising process. In flow matching, it learns a vector field that moves from noise toward the data distribution.

That sounds abstract, but the product consequence is straightforward: these models can be very good at naturalness, speaker similarity, rhythm, and controllability. You can condition on text, speaker information, duration, prosody, or style. The model is not trapped into producing one token after another in a long chain.

Training usually works by taking a clean speech representation, adding noise along a schedule, and teaching the model how to recover or move toward the clean target. Some systems use codec latents. Some use mel-like representations. Newer models often use transformer backbones adapted for diffusion-style generation.

The tradeoff is serving cost and streaming complexity. Multiple refinement steps are not free. A model that needs several denoising or flow-matching steps can be expensive at high volume. Streaming is also less natural than in a model that emits audio continuously, because the system often wants a chunk before it can refine and release it.

For many use cases, this is a good trade. If your priority is expressive speech, controllable prosody, or open-weights experimentation, this camp is very attractive. F5-TTS and E2-TTS made that clear for builders who want strong quality without being locked into a closed vendor.

For real-time phone calls, the open question is not whether the speech sounds good. It is whether the model can keep turn-taking tight when network hops, buffering, and telephony constraints are added.

State space models are built around streaming

State space models are the newest and smallest camp, but they are worth watching because they attack the problem that live voice products feel most directly: latency.

Cartesia Sonic is the visible production example. Cartesia's work is based on the Mamba family of state space models, connected to Albert Gu, Cartesia's co-founder and chief scientist.

A state space model does not use transformer attention in the usual way. It maintains a fixed-size hidden state as it processes a sequence. At each step, it updates that state and emits output. There is no attention over an ever-growing context window and no growing KV cache in the same transformer sense.

That matters for speech because live audio is sequential. A voice agent does not need to generate a perfect paragraph after a long wait. It needs to start responding quickly, keep the stream stable, and avoid awkward gaps.

A state-based architecture fits that job. The model can carry forward speaker and prosody information in its state while emitting audio as the state evolves. Memory stays bounded. Streaming is more natural because the system is not waiting to denoise a large chunk or decode a long token sequence before producing sound.

The tradeoff is ecosystem maturity. There are fewer open implementations, fewer engineers with direct experience, and less public detail than with transformer, VAE, or diffusion systems. Continuous representations can also be harder to inspect than discrete codec tokens.

This camp is not the default answer for every TTS product. It is the one to pay attention to when real-time streaming stability is the main constraint.

Foundation models make voice promptable

There is also a sixth category that does not fit cleanly inside the five speech-specific camps.

OpenAI's gpt-4o-mini-tts and the Realtime API are better understood as multimodal foundation-model systems. Voice is one modality inside a larger model that can handle text, audio, image, and reasoning tasks.

That changes the control surface. You are not cloning a specific voice from a sample. You are instructing behavior: sound calmer, sound like a tired professor, speak with more energy, slow down, be more empathetic. The model can respond to natural-language direction because speech generation is tied to a broader instruction-following system.

That is a real capability. It is useful when you want flexible voice behavior, fast experimentation, or speech-to-speech interaction where the same system handles input audio, reasoning, and output audio.

It is also a different tradeoff. If you need a branded voice, a voice double, audiobook continuity, or strict speaker identity from a sample, this is not the same tool as a cloning-focused TTS model. Cost and latency are tied to the larger model, not only to a speech-specific serving path.

This category matters because it points to where voice interfaces are going. In many products, TTS will not be a separate final step forever. It will be part of a live multimodal loop.

But for teams choosing a production TTS provider today, it should not be confused with classic custom-voice TTS. It solves a different problem.

The right architecture depends on what must not fail

No architecture wins every axis. The useful question is not which model sounds best in a sample. The useful question is what your product cannot afford to get wrong.

If you are building a real-time voice agent, latency and streaming stability dominate. Naturalness matters, but a beautiful voice that starts late will still make the conversation feel broken.

If you are building long-form narration, speaker consistency and editing control dominate. A slightly slower model may be fine if it preserves the voice over long passages.

If you are building high-volume transactional audio, cost and reliability dominate. The fanciest model may be the wrong one if every generated sentence burns margin.

The mistake is to collapse all of this into one benchmark. Latency, naturalness, controllability, multilingual coverage, cloning quality, streaming stability, and cost are separate axes. The architecture determines which axes the provider is naturally strong on and which ones it has to work around.

TTS architecture is only half the latency story

For production voice AI, the model architecture is not the whole system.

A TTS model that reports 90ms latency in isolation does not create a 90ms user experience if the audio travels across three vendors before it reaches the caller. In a stitched stack, audio may move from telephony to media streaming, then to STT, then to an LLM, then to TTS, then back through telephony. Every boundary adds network time, buffering, failure modes, and operational ownership questions.

This is why voice AI agents are not only a model-selection problem. They are an infrastructure problem.

Most teams choosing TTS are also choosing speech-to-text, an LLM, telephony, SIP connectivity, media streaming, and observability. The TTS architecture determines how the model behaves. The infrastructure determines how that behavior reaches the caller.

Those choices compound. A low-latency model deployed behind a fragmented call path can lose its advantage. A slightly slower model running close to the call path can feel faster to the user. A cloning-first model can be the wrong choice if the product really needs tight barge-in and predictable turn-taking.

That is the argument for AI agent infrastructure as a category. Voice AI does not break only because a model is weak. It breaks because audio crosses too many systems, state gets split across vendors, and nobody owns the full loop.

Telnyx runs voice, media streaming, STT, LLM routing, and TTS close to the carrier network where calls enter the system. That does not change the underlying TTS architecture. It changes whether that architecture can survive the production path.

When you choose a TTS provider, you are choosing more than a voice. You are choosing the failure mode your product will inherit.