---
title: What Is LLM Inference? How Prefill and Decode Work
canonical: https://telnyx.com/ai-inference/llm-inference-explained
---
# What Is LLM Inference? How Prefill and Decode Work

[View as page](https://telnyx.com/ai-inference/llm-inference-explained)

[Back to AI Inference](https://telnyx.com/ai-inference)

Last updated 16 Nov 2026

# LLM inference: how it works, from prefill to decode

What happens between sending a prompt and getting a reply, and why long documents, big contexts, and distant GPUs make you wait.

By [Eli Mogul](https://telnyx.com/authors/eli-mogul)Content Writer & Editor

What happens between sending a prompt and getting a reply, and why long documents, big contexts, and distant GPUs make you wait.

**Takeaways**

* Slow AI replies usually trace back to one of two moments: the model reading your prompt before anything appears, or the model generating the answer piece by piece.
* Memory, not raw GPU power, is usually what runs out first, because the model's notes on your conversation grow with every token.
* Short chats stay fast. Long documents and hours of piled-up history are where you start to feel the wait.
* Even on an API, you control the levers that matter most: prompt length, reply length, and how far your request travels.

## What is LLM inference?

Inference in an LLM is the step where a trained large language model runs on your new prompt and produces a reply. The weights stay fixed: training happens once, and inference happens on every request you send.

You've seen it plenty of times: you ask a chatbot where your package is, and there's a short pause before the answer starts to appear. That pause, and everything after it, is LLM inference at work.

### What happens when you send a prompt

LLM inference, sometimes called LLM inferencing, is a read-only [forward pass](https://telnyx.com/learn-ai/forward-propogation-ai): your prompt flows through the model's layers, and the model predicts the next token, a word or part of a word. The weights, the billions of numbers the model learned in training, don't change along the way.

That's the plain LLM inference meaning: a trained model applying what it learned to input it hasn't seen. Training is like writing the textbook, and inference is reading from it on every request. Whether you call it inference LLM work or a model call, the step is the same.

Language models are one case of [what is AI inference](https://telnyx.com/ai-inference/what-is-ai-inference) more broadly, which also covers speech, image, and classification models.

Here's why it matters to you: you pay for inference and you wait for it, on every single request. Training was someone else's bill. Inference is yours.

It costs energy too. Microsoft researchers estimate a median of 0.31 Wh per query for models over 200 billion parameters on H100 servers, rising about 13 times to 3.91 Wh for long reasoning queries with outputs about 15 times longer. Small per question, but it adds up fast across millions of them.

### Inference vs serving

Inference is the forward pass itself, and serving is the system that runs it for many people at once. You could run inference in a notebook on one GPU. Serving adds the endpoint, the request queue, the scheduler, batching, and autoscaling, the parts of an [inference pipeline](https://telnyx.com/ai-inference/inference-pipeline) that turn a model into an API.

Think of inference as the engine and serving as the whole car around it: the steering, the brakes, and the dashboard. When you call a hosted LLM API, most of the latency you can't see lives in that serving layer and the network in front of it.

## Prefill vs decode: the two phases of LLM inference

LLM inference prefill and decode compared

Every LLM inference request runs in two phases that stress the GPU in opposite ways: prefill is limited by compute, and decode is limited by memory bandwidth.

### Prefill reads the whole prompt at once

Prefill is the phase where the model reads your entire prompt in one parallel pass. A tokenizer first splits your text into tokens. Then every prompt token runs through the model together, and each token's attention keys and values go into the KV cache, the model's working memory of what it has already read.

The GPU does a lot of math for every byte it loads, so prefill is compute-bound: the chip's arithmetic speed is the limit. Prefill sets your time to first token.

You've probably felt this. Paste a 40-page contract into a chatbot and there's a noticeable pause before anything appears. Ask what time it is in Tokyo, and the reply starts almost before you've let go of the Enter key.

### Decode writes one token at a time

Decode is the phase where the model writes the reply one token per step, and each step reads all the model weights plus the whole KV cache. That's a lot of memory traffic for one token's worth of math, so decode is memory-bound: memory bandwidth, not arithmetic, sets the pace.

Picture a writer who rereads an entire reference book before adding each new word. Google's inference-scaling paper (Pope et al., MLSys 2023) puts numbers on it for one setup: a 500B+ parameter model with multihead attention, at batch size 512 and a 2,048-token context.

In that setup, the KV cache reaches about 3 TB, three times the size of the model's parameters, and the chip sits essentially idle while it streams that memory. Decode sets how fast tokens stream, which is why a reply can start quickly and still take a while to finish.

If you've ever watched an answer start right away and then trickle out for what feels like forever, you've watched decode.

Here's how the two phases compare.

_Prefill vs decode in LLM inference._

|                        | Prefill                                       | Decode                                                  |
| ---------------------- | --------------------------------------------- | ------------------------------------------------------- |
| What it does           | Reads every prompt token in one parallel pass | Writes one new token per step                           |
| What limits it         | Compute (math per second)                     | Memory bandwidth (bytes read per second)                |
| What it fills or reads | Fills the KV cache                            | Reads all the weights and the whole KV cache every step |
| What it sets           | Time to first token                           | How fast tokens stream, and total time                  |
| Grows with             | Prompt length                                 | Output length                                           |

### Does a longer conversation slow down LLM inference?

A longer conversation slows LLM inference only once the context gets large. Every turn resends the whole history, but a few hundred extra tokens barely move prefill time.

In a multi-turn benchmark Telnyx ran and [published on GitHub](https://github.com/team-telnyx/benchmark/tree/main/inference/2026-10-multi-turn-providers), the prompt grew from a median 122 tokens at turn 1 to 1,141 at turn 4\. Median time to first token stayed at about half a second throughout, from 0.48 to 0.50 seconds.

That run covered three open-weight models and 2,129 completed turns, with four providers pooled. It didn't record whether providers cached the shared history, so caching may be part of why the early turns stay flat.

Prefill time shows up once the context reaches thousands of tokens. In a second Telnyx run, a [prompt-size benchmark](https://github.com/team-telnyx/benchmark/tree/main/inference/2026-09-glm-5-3-fresh-prompts), median time to first token was 0.44 seconds at about 900 prompt tokens, 1.00 second at about 9,500, and 4.77 seconds at about 89,000.

LLM inference time to first token

Treat both results as directional. In practice, a quick support chat stays snappy. Paste in a long document, let a voice call run for an hour, or feed in a big tool output, and you'll feel the wait before the first word.

The fix is caching the shared history, called prefix caching, so the model doesn't reread what it has already seen.

## The KV cache: why LLM inference is a memory problem

The KV cache is the memory an LLM keeps of every token it has already processed, so it never has to recompute them. It also grows with every token and every user, which is why GPU memory, not math, usually caps how many requests a server can run at once.

### What the KV cache stores

The KV cache in an LLM stores the attention keys and values for every earlier token, one set per layer. Attention is how the model decides which earlier tokens matter for the next one, and keys and values are what it compares and pulls from.

It's like keeping notes during a meeting instead of replaying the whole recording every time someone speaks. Each new token only needs its own math, plus a lookup into the notes.

### How big the KV cache gets

The KV cache grows by a fixed amount for every token, and that amount is large. In the PagedAttention paper (Kwon et al., SOSP 2023), OPT-13B needs 800 KB of KV cache per token, so a single request can reach 1.6 GB.

You can work out the size for any model: 2 (keys and values) x layers x KV heads x head size x 2 bytes at 16-bit precision. Meta's published architecture for Llama 3.1 70B lists 80 layers, 8 key-value heads, and a head size of 128.

By our calculation, that's 2 x 80 x 8 x 128 x 2 = 327,680 bytes, or 320 KB per token. A full 128K context (131,072 tokens) needs about 40 GB of KV cache for one request, on top of about 140 GB of weights.

Now imagine a few dozen people holding long conversations on the same server at once. You can see why memory runs out long before the math does.

### How servers stop wasting KV cache memory

Servers stop wasting KV cache memory by storing it in small fixed-size blocks that are allocated as a sequence grows, instead of reserving room for the longest possible reply up front. This technique is called PagedAttention.

The same paper found that earlier systems used only 20.4% to 38.2% of KV cache memory for actual token states. vLLM, a widely used open-source serving engine, is built on this approach.

Make your first inference call

Create an API key in the Mission Control Portal and point your code at Telnyx.

[Sign up](https://telnyx.com/sign-up)[Contact us](https://telnyx.com/contact-us)

## Batching: how LLM inference serves many users at once

Decode leaves most of the GPU's math idle, so servers fill it by running many users' requests in the same step. One read of the weights then serves dozens of requests instead of one. Think of it as carpooling: the GPU makes one trip through the weights and drops off a token for everyone on board.

### Static vs continuous batching

Continuous batching in LLM inference swaps finished requests out and new ones in after every decode step, while static batching waits for a full batch and runs it to the end. With static batching, a short request gets stuck behind the longest one in its group.

Continuous batching works more like a restaurant that seats new guests as soon as a table opens. The idea comes from Orca (Yu et al., OSDI 2022), which introduced iteration-level scheduling. Orca reported 36.9 times the throughput of NVIDIA FasterTransformer on GPT-3 175B at the same latency level.

The trade-off: bigger batches raise total tokens per second, but they can raise each person's wait.

### Why the GPU sits idle

A GPU can sit idle even with batching, because the CPU handles tokenizing, scheduling, sampling, and detokenizing between GPU steps. If that work runs one request at a time, the GPU waits on every token. Picture a fast chef in a kitchen with one server: the food is ready in seconds, but orders arrive one at a time.

To tell this apart from a slow model, watch GPU and CPU use together under load. Low GPU use next to a maxed-out CPU core points at the CPU, not the model.

### Other ways to speed up LLM inference

Beyond batching, three common techniques each target a specific phase. Prefix caching speeds up prefill by computing a shared prompt, such as a long system prompt, once and reusing it.

Quantization speeds up decode by storing weights at lower precision, such as 8 bits instead of 16, so each step reads fewer bytes.

Speculative decoding speeds up decode too: a small draft model proposes tokens, and the full model checks them in one pass. None of them is free, so test each one on your own prompts before you ship.

## How LLM inference speed is measured

Four numbers describe how LLM inference feels to your users: time to first token, time per output token, total tokens per second, and tail latency.

### TTFT, time per output token, and tokens per second

[Time to first token](https://telnyx.com/ai-inference/ttft) (TTFT) is how long you wait before the first word appears, and it includes the network, the queue, and prefill. Time per output token is the gap between tokens once the reply starts streaming.

Total tokens per second measures output across every user on a server. It's a better capacity number than requests per second, because requests vary so much in length.

Tail latency, the p95 or p99, covers your slowest 5% or 1% of requests. Averages can look fine on a dashboard, but nobody remembers the average reply. They remember the one that hung.

### What counts as fast

What counts as fast depends on the use case, and MLPerf gives one reference point. In MLPerf Inference v5.1, MLCommons set its Llama 3.1 8B benchmark at a TTFT of 2 seconds or less and a time per output token of 100 ms or less for the server scenario.

The interactive scenario tightens that to 0.5 seconds and 30 ms. Other MLPerf models have different targets. Our guide to reading [inference benchmarks](https://telnyx.com/ai-inference/inference-benchmarking) covers how to apply numbers like these to your own traffic.

### Why benchmark numbers disagree

Benchmark numbers disagree because client overhead, prompt shapes, and concurrency all change the result. Even the measuring tool matters: a 2026 Google paper shows that Python benchmark clients, held up by the global interpreter lock, inflate measured TTFT and time per output token.

Compare first-token and full-response numbers side by side, too, since a fast first token can hide a slow finish. Before you blame the model, check the stopwatch.

## Running LLM inference yourself or through an API

Even if you only call an API, the mechanics above decide your bill and your latency: prompt length sets prefill time, output length sets decode time, and the distance to the GPU adds to both. The good news: you have more control here than you might think.

### What self-hosting LLM inference takes

Self-hosting LLM inference means running a serving engine on GPUs you rent or own, and handling batching, memory, and scaling yourself. If you're wondering whether you need your own GPUs, the honest answer is usually not yet.

A Carnegie Mellon cost study (Pan et al.) finds on-premise deployment makes sense at about 50 million tokens a month or more, or when strict data residency rules out an API.

Those results assume hardware running 8 hours a day, 20 days a month, electricity at $0.15 per kWh, and no staffing costs. The same paper calls 10 to 50 million tokens a month the sweet spot for medium models, so treat 50 million as a guide, not a floor.

### How LLM inference APIs charge

LLM inference APIs charge per million tokens, with input and output tokens priced separately and output usually priced higher. So a long prompt and a long reply cost you in different ways, the same split you saw between prefill and decode. Trimming a bloated system prompt and asking for tighter answers both show up on your invoice.

For current numbers and the levers that lower them, see our breakdown of [LLM inference cost](https://telnyx.com/ai-inference/inference-costs).

### Why the GPU's location matters

The GPU's location matters because every request crosses the network twice, once to reach the GPU and once to bring the reply back. A GPU far from your users adds latency before prefill even starts. Someone in Frankfurt shouldn't wait on a GPU in Virginia, however fast that GPU is.

"We often obsess over LLM inference speeds, but in Voice AI, the network is often the silent killer." **Ian Reither, COO at Telnyx**

Telnyx's [Inference API](https://telnyx.com/products/inference) serves open-weight LLMs on GPU infrastructure Telnyx owns, through an OpenAI-compatible Chat Completions API. Telnyx runs inference GPUs in five regions on four continents: calling a regional endpoint sends your request to the nearest GPU region, and a request can also pin a specific region. For interactive work on select models, the Priority service tier trades a higher token price for lower latency.

### Frequently asked questions about LLM inference

Quick answers to common questions about LLM inference.

**How much does LLM inference cost?**  
LLM inference through an API is priced per million tokens, with input and output tokens billed separately. Rates vary with model size and provider, and larger models usually cost more per token. Self-hosting swaps token prices for GPU, power, and operations costs, which pays off mainly at high, steady volume. For current numbers, see our guide to [LLM inference cost](https://telnyx.com/ai-inference/inference-costs).

**Can you run any open-source model through an inference API?**  
No. An inference API serves the models in its catalog, so running a model outside that list means hosting it yourself or using a provider's dedicated deployment. Catalogs change often, so check before you build. Telnyx lists the open-weight models it currently serves in its [hosted model list](https://developers.telnyx.com/docs/inference/models).

**What is speculative decoding in LLM inference?**  
Speculative decoding speeds up decode by pairing the full model with a small draft model. The draft proposes several tokens, and the full model checks them in one pass, keeping the ones it agrees with. Google researchers (Leviathan et al., ICML 2023) showed the output matches the full model's, and decode gets faster whenever the draft is usually right.

**What is the best LLM inference engine?**  
There's no single best LLM inference engine. The right one depends on your hardware, your model, and whether you need maximum throughput or the lowest latency per user. vLLM, SGLang, and TensorRT-LLM are the most widely used options, and each makes different trade-offs, so test them on your own traffic before you commit.

**Are open-weight models a step down from frontier models?**  
Not necessarily: the gap depends on the task. For many tasks, such as summarization, extraction, and customer support, a well-chosen open-weight model performs close to frontier models, and its lower latency and cost can matter more. For the hardest reasoning tasks, test both on your own prompts. Our guide to [open-weight models](https://telnyx.com/ai-inference/open-source-models) covers how to choose.

### Put what you just learned to work

Run open-weight LLMs on GPUs Telnyx owns, through an OpenAI-compatible API that routes your requests to the nearest of five GPU regions.

[**Start building**](https://telnyx.com/products/inference)

Eli Mogul

Content Writer & Editor

Eli is the content writer and editor at Telnyx. Born and raised in Chicago, Eli attended the University of Missouri where he obtained a BA in Journalism. Eli joined Telnyx in August of 2025\. In his spare time, you'll find Eli reading, playing video games, or running.

[Start building](https://telnyx.com/sign-up)