Inference optimization: cut LLM latency and cost

What is inference optimization and why does it matter?

Inference optimization is the practice of improving the latency, throughput, and cost per token of a trained model once it serves production traffic. Training is a one-time bill, inference is the one you pay on every request, and the work happens on three layers: the model (quantization, distillation, pruning), the serving infrastructure (runtime, batching, cache management, placement), and the request path, meaning the queue and the network between your app and the GPU.

Most guides stop at the first two layers. The third is where a model that benchmarks well still times out.

Prefill and decode: two phases, two different bottlenecks

Every request to a large language model runs in two phases, and LLM inference optimization starts with knowing which one you are waiting on:

• Prefill: the model processes your whole prompt at once as a matrix-matrix operation, so the GPU's compute units stay busy.
• Decode: the model produces one token per step, and each step reads the weights and the cached keys and values back from GPU memory.

That makes decode memory-bound. The arithmetic per step is cheap and the data movement is what you wait on, so a faster GPU helps decode less than moving fewer bytes does. Almost every technique in the next section targets decode for that reason, and it is why your machine learning inference bill tracks memory bandwidth more closely than peak compute.

The metrics to record before you change anything

AI inference optimization only works against a baseline. Record these five numbers first, and read TTFT vs end-to-end latency for how they differ in practice:

• TTFT (time to first token): prefill plus queueing. It is the first delay your chat and voice users feel.
• Time per output token: decode speed, memory-bound. It sets how fast an answer streams.
• End-to-end latency: TTFT plus every output token plus the network. Any request with a timeout is judged on it.
• p99 latency: the slowest 1% of requests. It is what breaks SLAs and agent tool loops.
• Cost per token: GPU cost divided by throughput. It is the number on your inference bill.

These metrics pull against each other. Bigger batches raise tokens per second, but every request waits longer to start, so TTFT rises. You pick an operating point under a latency target, and maximum throughput is rarely that point. Watch p99 and end-to-end latency, not averages, if you want to cut inference latency your users notice.

End-to-end latency is the metric to track because the model is only one stop on the path. As one Telnyx engineer puts it:

Compare tokens per second only between models that share a tokenizer. A different tokenizer turns the same text into a different number of tokens, so its tokens per second is a different unit.

Measure open-weight models before you tune them

The Telnyx Inference API serves open-weight models on in-region GPUs behind OpenAI-compatible endpoints, so you can record TTFT and p99 first.

Try Telnyx Inference

LLM inference optimization techniques that move the numbers

The LLM inference optimization techniques that matter either cut the bytes decode has to move or cut the work each token needs, and every one of them costs you something in accuracy, memory, or latency.

Which optimization fits your bottleneck?

• Answers stream slowly, long outputs: quantize weights and use a paged KV cache. Every decode step rereads weights from GPU memory, so smaller weights stream tokens faster.
• TTFT climbs when traffic is heavy: cap batch size or pick a latency-first service_tier. Bigger batches add tokens per second but make each request wait longer to start.
• Same long system prompt on every call: keep the prompt prefix fixed on Telnyx Inference. Cached input costs up to 88% less and skips repeated prefill work.
• Model benchmarks fast, p99 still times out: serve on in-region GPUs with Telnyx Inference. Queueing and cloud boundaries the request crosses, not the model, usually set p99 and end-to-end latency.
• Offline batch jobs, nobody waiting: push batch size up with continuous batching. With no latency target, maximum throughput gives the lowest cost per token, since cost is GPU spend over throughput.

Quantization and the KV cache: the memory math

GPU memory holds two things that decide how much traffic you can serve: the weights and the KV cache. In FP16 each parameter takes 2 bytes, so a 7B model needs about 14 GB for weights alone. The KV cache stores keys and values for every token in every layer, so its size per token is 2 x layers x hidden size x bytes per value. For a Llama 2 7B class model with 32 layers and a hidden size of 4,096, that is about 0.5 MB per token, or about 2 GB at 4,096 tokens and batch 1. Batch 2 doubles it, because the cache grows linearly with batch size and sequence length.

Each term has its own fix. Quantization shrinks the weights: weight-only INT8 or INT4 cuts the bytes decode reads on every step, while activations are harder to quantize because of outlier values. Post-training quantization (PTQ) is cheap, and quantization-aware training (QAT) holds accuracy better at the price of a training run. FP8, INT4, AWQ, and GPTQ are the common formats, and each needs an evaluation on your own prompts before it ships. Grouped-query attention (GQA) and paging shrink the cache instead, and Llama 2 70B uses GQA for that reason.

PagedAttention and continuous batching: filling the GPU

Older servers reserved one contiguous cache slot per request at its maximum length, so a 2,048-token slot for a 50-token answer wastes almost all of it. The PagedAttention paper names three kinds of waste: reserved space, internal fragmentation, and external fragmentation. PagedAttention removes them by storing the cache in fixed-size blocks tracked by a block table, the way an operating system pages memory.

Continuous batching, also called in-flight batching, spends that freed memory. Static batching makes every request wait for the longest one in its batch. Continuous batching evicts finished sequences and admits new ones at every step, so the GPU stays full and tokens per second per GPU rises.

The throughput-vs-latency rule: Doubling the batch roughly doubles tokens per second per GPU until memory runs out, but every request now waits for the batch to form, so TTFT rises. Set the batch size from your latency target, then buy throughput with more replicas or a different serving tier.

Speculative decoding and prefix caching: doing less work per token

Speculative decoding runs a small draft model that proposes several tokens, and the target model verifies them in one parallel pass. When the draft guesses right, you get several tokens for the price of one decode step. Three things decide whether it pays off:

• Acceptance rate: a draft the target often rejects adds work instead of saving it.
• Batch fullness: the gain shrinks when the batch already keeps the GPU busy.
• Memory headroom: the draft model has to fit beside the target and its cache.

Prefix caching skips work you already did. It keeps the KV cache for a shared system prompt, tool schema, or conversation history and reuses it on the next request, so prefill runs only on the new tokens. For e-commerce AI agents and other agents that resend the same system prompt and tools on every call, that repeated prefill is a large share of the input bill.

Tensor and pipeline parallelism come in when the model does not fit one GPU. Both split it across devices to cut per-device memory, both add interconnect cost, and pipeline parallelism leaves idle bubbles while stages wait on each other. Find your bottleneck in the table and read across.

LLM inference optimization techniques by bottleneck and cost

Apply them in order: record a baseline, quantize the weights, page the KV cache, batch continuously, then pick a serving tier or add replicas. Each step changes one number you can check against the last.

vLLM inference optimization and the llama.cpp equivalents

vLLM and llama.cpp are the two engines most teams run, and they apply these techniques for different workloads.

What vLLM gives you out of the box, and what you still tune

vLLM is built around PagedAttention and continuous batching, so the first win is on by default. It also supports tensor parallelism across GPUs. Our guide to vLLM inference explained covers the engine itself.

What you still tune are the operating-point levers: the share of GPU memory given to the KV cache, the maximum batched tokens and concurrent sequences, the quantization format of your checkpoint, prefix caching, and speculative decoding. Every one trades latency for throughput. More cache memory allows bigger batches, and bigger batches raise TTFT. The exact flag for each lever is in the vLLM documentation, which tracks the current release.

Change what you measured first: Giving the KV cache more memory raises throughput and TTFT together. If your complaint is p99 timeouts, look at queueing and batch limits before you quantize anything.

Where llama.cpp fits: quantized weights on CPUs and small GPUs

llama.cpp runs quantized weights on CPUs, Apple silicon, and single consumer GPUs. llama.cpp inference optimization is mostly two choices, the quantization level of the weight file and how many layers you offload to the GPU, plus context length and thread count. The names for each are in the llama.cpp repository README. It suits edge inference, local tools, and low-concurrency work where the model has to fit a small device. Side by side, the two engines split like this:

• vLLM: continuous batching, PagedAttention block allocation, pre-quantized checkpoints, and tensor parallelism across GPUs. Best for many concurrent users on server GPUs.
• llama.cpp: single or small batches, a contiguous cache sized by context length, a quantization level chosen per weight file, and layer offload to one GPU. Best for local, edge, and low-concurrency use.

TensorRT-LLM, SGLang, and other engines make different tradeoffs again, and we cover them in inference engines compared.

When self-hosting stops paying for itself

Self-hosting pays while your GPUs stay busy and your team has time to maintain the stack. Three signals tell you it has stopped:

• Upgrade overhead: engineers spend weeks each quarter on new engine versions and new model releases.
• Idle GPUs: your cards sit unused between traffic bursts, and you still pay for them.
• Unexplained timeouts: p99 spikes show up that your own metrics cannot trace to a cause.

If two of them show up over a month, compare the same open-weight model behind an OpenAI-compatible managed endpoint, where autoscaling from zero takes the idle-GPU problem off your books.

Which is the best platform for AI inference optimization?

The best platform for AI inference optimization lets you choose your latency-vs-throughput operating point per request instead of re-tuning a cluster, and Telnyx Inference is built that way. It serves a curated library of open-weight models, including models from z.ai, DeepSeek, and Minimax, as serverless inference on GPUs Telnyx owns, with OpenAI-compatible chat completions, so you change the base URL instead of rewriting code.

It costs up to 75% less than proprietary alternatives, with no GPU rental fees or minimums, and it supports structured output through JSON mode. The hosted model list shows the current lineup.

Managed API or your own stack: the decision rule

The choice comes down to who should own the inference infrastructure, and a short rule settles it:

• Self-host when you need a model no managed provider hosts, need custom kernels or a fine-tuned checkpoint you cannot upload, or run steady load that keeps your own GPUs at high utilization.
• Use a managed API when your traffic is bursty, the model you want is in the hosted lineup, data residency requires in-region compute, or your team's time is better spent on the product.

Pick the operating point per request with a serving tier

On Telnyx, the same hosted model and context window run on different serving capacity, so the only things that change between tiers are latency and price. You choose with the service_tier field on each request:

• Default: standard capacity at standard rates. Every published benchmark and price comparison refers to this tier, and it is what you get if you omit service_tier.
• Priority: faster serving capacity for low-latency interactive workloads, including voice orchestration for Telnyx Voice AI and contact center AI. It is live on select models today, with more models and regions coming.
• Flex: the lowest rates, for background work such as offline evaluation, document processing, and summarization where minutes of end-to-end latency are acceptable. Requests stay synchronous, and Flex is available in the US on select models.

That turns the batch-size tradeoff into a request field. Test interactive traffic on Priority, and compare providers on Default so the numbers match published benchmarks. Cached input is up to 88% off, which puts a price on prefix caching, and our guide to inference cost optimization shows how it adds up alongside the rates on inference API pricing.

The request path is the layer most optimization work skips. Telnyx Inference runs in-region across the Americas, Europe, MENA, and APAC, with LATAM coming soon, so your data stays where your users are, with zero data retention and no training on your data. There are no inter-provider hops between your request and the model, and the service autoscales from zero to thousands of requests per second with no cold starts. Read more on running GPUs on the network.

Send one request to the Telnyx Inference API

This is an OpenAI-compatible chat completion sent to the Telnyx endpoint with a model name and one message. To choose a serving tier, add the service_tier field to the request body.

The model in the example is covered in Kimi K3 on Telnyx, and the AI gateway docs show how to route existing OpenAI SDK code to the same endpoint. From here you can name your bottleneck and pick your fix: tune it yourself in vLLM or llama.cpp, or change one field on a Telnyx request.

Inference optimization FAQ

How much can moving from a closed frontier model to an open-weight model cut our inference cost?

It depends on the model pair and your token mix, but the gap is large. Telnyx Inference lists open-weight models at up to 75% less than proprietary alternatives, and cached input at up to 88% off. Agents that resend the same system prompt on every call gain the most, because that repeated input is the part cached pricing discounts.

Why does the same open-weight model time out or run slower on one provider than another?

The weights are the same, but the serving stack is not. Providers choose different batch sizes, cache memory, quantization formats, and capacity, and requests cross different networks to reach the GPU. Bigger batches raise throughput and TTFT together, so a provider tuned for throughput can show p99 spikes your timeout catches. Compare providers on p99 and end-to-end latency, not average tokens per second.

Are open-weight models good enough to replace a closed model for production AI agents?

For many agent workloads, yes, but test it on your own traffic before you switch. Run your real prompts, tool calls, and output formats through the open-weight model, check accuracy, schema adherence, and latency against your current model, and move traffic in stages. Structured output features such as JSON mode help keep tool calls and formats consistent.

What is inference optimization?

Inference optimization is the work of cutting latency, raising throughput, and lowering cost per token when a trained model serves production traffic. It covers the model (quantization, distillation, pruning), the serving stack (batching, cache management, parallelism), and the request path between your app and the GPU. The right technique depends on which of those is your bottleneck.

Is inference optimization still worth doing, or will faster GPUs make it irrelevant?

It stays worth doing. Decode is limited by how fast memory moves bytes, and that improves more slowly than compute, so a faster GPU helps less than quantization or cache paging. Models and context windows also keep growing, which raises memory demand. And no GPU removes queueing delay or a network hop, so the request path still needs work.