What is AI inference? How a trained model turns input into answers

What is AI inference?

AI inference is an LLM doing the job it learned in training. Think of a radiologist. Years of medical school and residency are the training. Every scan they read on a Monday morning is inference: new images, the same knowledge, and a diagnosis that follows.

AI works the same way. Your support bot spent its training learning language from huge amounts of text. When a customer's ticket arrives, the bot reads it and writes a two-sentence summary. The next ticket is another inference, and so is the one after that.

That's why inference is where you pay. The model's maker pays for training, once per model. You pay for inference on every ticket, in money, waiting time, and energy, for as long as the bot runs.

What does inference mean in AI?

In AI, inference means running a finished model on data it hasn't seen before to get an output: a summary, a label, a fraud score, a reply.

What the model learned in training lives in its weights, the millions or billions of numbers it adjusted while studying its data. Inference reads those weights and never changes them. That's the difference between the radiologist reading a scan and the radiologist in training.

Why statisticians use inference differently

In statistics, inference means using data to draw conclusions, such as whether a new drug really works or the trial result was just luck. When you run a finished model on new data, statisticians call that prediction, not inference.

A University of Berkeley statistics course from Fall 2026 says so on one slide: "This class will study prediction, which is different from inference."

So when a statistician says a model needs better inference, they don't mean faster GPUs.

How AI inference works: step by step

Every AI inference request runs the same three steps, regardless of the underlying model.

Here's how it applies for for a fraud model checking your card payment:

1. The app prepares the input. It turns your payment into numbers the model can read: the amount, the merchant, the time, and where your card was last used.
2. The model runs once on those numbers. Engineers call that single run a forward pass.
3. The app gets the output back. The model returns a score, say a 2% chance of fraud, and the bank approves your payment before the terminal beeps.

A chatbot runs the same three steps with one twist: step two repeats once for every token, a piece of a word, that it writes. Your text becomes tokens, the model reads them all at once, and then it writes the answer one token at a time.

Let's say you ask a model to summarize a support ticket in two sentences, with a customer's ticket pasted underneath. Here's how the inference request would work:

Step 1: the prompt becomes tokens

A tokenizer splits your prompt into tokens, pieces of words that each map to an ID in the model's vocabulary. A short common word is often one token. A long or rare word, or the customer's order number, can be several.

Step 2: prefill reads the whole prompt at once

Prefill is the first phase of LLM inference: the model processes every prompt token in parallel. It saves its intermediate results for each token in a cache (the KV cache), so later steps don't redo that work.

Because prefill runs in parallel, a GPU gets through thousands of prompt tokens fast. Your 900-token ticket is a light load.

A longer prompt still means more work: in the benchmark below, a prompt 95 times longer waited about 11 times longer for its first token.

Prefill ends when the model produces its first output token.

Step 3: decode writes the answer one token at a time

Decode is the second phase: the model picks one token, adds it to the sequence, and runs again to pick the next. Each token waits for the one before it. Your two-sentence summary might be 50 tokens: "The", then "customer", then "paid", one step at a time.

Google researchers describe it in one line: decoding K tokens takes K serial runs of the model.

In a benchmark Telnyx ran and published on GLM 5.3, an open-weight model anyone can download, output arrived at median rates of 89 to 124 tokens per second.

That's roughly 8 to 11 milliseconds per output token, so the 50-token summary takes about half a second to write.

How long does an inference request take?

Prompt length and output length determine how long an inference request takes from start to finish.

The support ticket summary example sits close to the benchmark's short prompt. A 935-token prompt with 120 output tokens got its first token after a median 0.44 seconds and finished at 1.66, so expect the summary in under two seconds.

Now paste in the customer's whole account history. A prompt about 95 times longer, 88,727 tokens, pushed the first token to 4.77 seconds and the finish to 6.20. That's why a chatbot that starts answering in under a second in a demo can take several seconds once customers paste their whole email history.

A short prompt with 10 times the output, 1,200 tokens, still got its first token at 0.47 seconds but finished at 12.64.

Your prompt sets how long you wait for the first token. Total time is that wait plus the time to write every output token, so output length drives most of it once answers get long.

For provider-by-provider results, see the TTFT vs E2E benchmark. Published inference benchmarks help you compare providers, but your own prompts are the real test.

AI inference vs training

Training changes a model's weights; inference leaves them alone and runs the model as it is. That one difference shapes everything else: the data each one uses, how often each one runs, and how each one shows up on your bill.

Training vs inference at a glance.

Inference in generative AI vs classic machine learning

Inference in generative AI works differently for text: a large language model runs once per token it writes, while a classic model, like a spam filter, runs once per input.

One pass per input vs one pass per token

A classic model makes one forward pass per input: one run from input to output, ending in a label or a score. A spam filter, a standard case of machine learning inference, reads an email once and says "spam" or "not spam."

A language model repeats the decode loop from step 3, so its time and cost grow with the length of what it writes.

Why reasoning models do more work at inference time

Reasoning models do more work at inference time because they write out their thinking as tokens before the answer. Every thinking token is another run of the model. Since it counts toward your output tokens, the thinking can cost more than the answer itself.

Snell and colleagues found that extra work at inference time buys accuracy. Searching over and revising candidate answers let a smaller model beat one 14 times its size, for the same total compute.

Types of AI inference: real-time and batch

AI inference runs in one of two ways, depending on who is waiting: a person right now (real-time) or nobody (batch). Where the model runs is a separate choice.

Real-time inference

Real-time inference, also called online inference, answers one request while a person or system waits. A chat reply, a fraud check at checkout, and a voice agent's next sentence all run this way.

Latency is what matters most in real-time inference. That's why chat apps stream the reply: they send each token as soon as the model writes it, so you see text before the model finishes.

Batch inference

Batch inference runs large jobs where nobody waits for a single result: overnight document classification, embeddings for a search index, and model evaluations. Throughput matters more than latency in batch inference. Providers usually price batch work below real-time requests.

Where AI inference runs: cloud, edge, and on device

AI inference runs in one of three places: a cloud data center, an edge server near your users, or the user's own device, such as a phone. The closer it runs to the user, the shorter the trip each request makes, and the smaller the model has to be.

Cloud inference

Cloud inference gives you the biggest models, called over an API from a data center. The trade-off is the network trip each request makes. A user in Dubai calling a model hosted in Virginia waits for that round trip before prefill even starts.

Large-model inference runs on accelerators: GPUs, Google's TPUs, and other inference hardware built for it.

Edge and on-device inference

Edge inference runs on servers near your users, and on-device inference runs on the phone or laptop itself. Both cut network time and keep data closer to home.

Running close to the data also keeps the app working when the connection drops. It also sends less data over the network, which cuts bandwidth costs.

Apple's on-device foundation model has about 3 billion parameters, trained to work on a phone.

Compare that with the 671 billion parameters of DeepSeek V3, which runs in data centers. That's more than 200 times Apple's model.

AI inference costs

AI inference costs input tokens times the input price, plus output tokens times the output price. Output tokens usually cost more.

For example: a help desk that handles 10,000 tickets a day. At about 900 tokens in and 50 out, that's 9 million input tokens and 500,000 output tokens daily. Multiply by your provider's per-million prices, and you have the bill, before any thinking tokens.

Paste in each customer's account history instead, about 89,000 tokens, and the same 10,000 summaries need roughly 887 million input tokens a day. That's about 100 times the input bill.

At $1.25 per million input tokens and $4.00 per million output tokens, the ticket-only summaries cost about $13 a day. With the account history, they cost over $1,000 daily.

Price your own token counts in the calculator on the Telnyx inference pricing page.

An inference bill grows with three things: more requests, longer prompts, and longer outputs. Requests can grow fast. Alphabet said in October 2026 that its model APIs process nearly 32 billion tokens per minute, more than four times a year earlier.

Check out Telnyx's overview on inference cost to track current prices and ways to cut them.

What makes inference fast or slow

Inference latency, the time a request takes, starts with the model and where it runs. A bigger model does more math per token, and faster GPUs process that math quicker. A busy provider queues your request, and distance adds network time to every round trip.

Inference API or your own GPUs

Inference as a service is the fastest way to start: send requests to an API, the provider runs the model, and users are billed per token. Running your own inference engine, the software that loads a model onto GPUs and answers requests, buys control at the cost of operating it.

Your own GPUs cost the same per hour whether they're busy or idle, so they pay off only when you keep them busy. If your ticket summarizer runs only during an eight-hour workday, those GPUs sit idle the other 16 hours. An API charges nothing for those.

Telnyx's Inference API serves open-weight models on GPUs Telnyx owns, through OpenAI-compatible endpoints, in the Americas, Europe, MENA, and APAC. Each request can set a service tier, priority, default, or flex, to trade speed for price without changing models. Telnyx also publishes its inference benchmarks, so you can check the numbers yourself.

AI inference FAQs