Inference cost explained: why your AI bill rises as prices fall

What is inference cost?

Inference cost is what you pay to run data through a trained model and get an output back. Hosted APIs bill it per token, and your own hardware bills it per GPU-hour. If you want the mechanics first, start with how LLM inference works.

Telnyx Inference uses per-token billing on hosted open-weight models. There are no GPU rental fees or compute surcharges, and no minimums. Per-token rates vary by model and service tier.

Every per-token price rests on GPU economics. Your cost per token is the GPU-hour price divided by the tokens that GPU produced in that hour, so a GPU that sits idle for half the hour makes each token it does produce twice as expensive. That is why the inference hardware you pick and how busy you keep it decide whether self-hosting ever pays off. The FAQ at the end of this blog covers the breakeven.

What the per-token meter actually counts

Your invoice applies four rates to four kinds of token. Input is everything you send: the system prompt, conversation history, retrieved documents, and the user's message. Output is everything the model writes back. Cached input is a prompt prefix the provider has already processed and reuses at a discount. Batch or low-priority tokens are work you let the provider schedule, billed below the standard rate.

Each line on your bill maps to one of these rates.

The rates on every inference bill

Reasoning tokens are the line you never see. A reasoning model bills its thinking as output, so a two-sentence answer can carry a long and expensive preamble. The rate card sets your floor, and four things decide how far above it your bill lands:

• Token volume: how many tokens you send and receive each month.
• Model choice: which rate card every one of those tokens is priced on.
• Context length per call: the history and retrieved documents you resend on every request.
• Calls per user action: how many model calls one click or message triggers.

Why output tokens cost more than input tokens

Output costs more because the GPU does more work for each output token. During prefill, the model reads your whole prompt in parallel, and the work is limited by raw compute. During decode, it generates one token at a time, and each step waits on memory bandwidth to move the model's weights. A GPU writes far fewer tokens per second than it reads.

Across providers, that gap shows up as output priced at 3 to 5 times input. The exact ratio varies by model, so check the rate card for the model you run. For your bill, it means a verbose answer costs more than a verbose prompt, and capping output length is a pricing decision.

Inference cost vs training cost

Training a model is a one-time cost, while inference recurs on every request and grows with every user you add. A successful product's cumulative inference bill overtakes its training bill, and industry estimates cited by NVIDIA put the production split near 80% inference to 20% training.

Check the rate card: See the current per-token rates for hosted open-weight models on the Telnyx inference pricing page.

AI inference cost trends 2026: why bills rise as prices fall

Token prices for a fixed level of capability have collapsed, and most inference bills still go up. Both are true because the rate card and the bill measure different things: the rate card prices one token, and your bill counts how many tokens your product now burns per user.

Token prices keep collapsing

The Stanford AI Index 2025 found that the cost of querying a model at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. If you priced a feature on a 2023 rate card, the same capability now costs a small fraction of that.

Why your bill still grows faster than revenue

Your bill grows because your traffic shape changes faster than rates fall. An agent makes 10 to 20 model calls for a single user action, where a chatbot made one. Context per call inflates 3 to 5 times when you resend untruncated history and retrieved documents. Reasoning tokens bill at the output rate, the most expensive line on the card.

These multipliers stack. Ten calls per action, each carrying four times the context, adds up to roughly 40 times the input tokens of a single-call chatbot on the same rate card. Teams building e-commerce AI agents feel this first, because every order lookup, return, and recommendation runs several calls deep. Our guide to scaling AI cost shows how those multipliers compound as usage grows.

What changed in 2026: Cached-input discounts of 90% became standard at closed providers. Agent call multiplication became the main driver of inference spend. Open-weight models became available on hosted, OpenAI-compatible endpoints at up to 75% less than proprietary alternatives.

Open-weight models still trail closed models on video and multimodal work, so the first traffic to move is text.

How providers like OpenAI cut inference cost, and how to get the same cuts

Closed providers lower the price you pay with four mechanics: cached input, batch processing, smaller models, and serving engineering. All four exist on open-weight models hosted on Telnyx, starting from a lower rate card.

Cached input, batch APIs and smaller models

OpenAI caches prompt prefixes longer than 1,024 tokens automatically, and its newest flagships bill cached reads at up to 90% off, according to OpenAI API pricing. Anthropic bills cache reads at 10% of the input rate. Both run batch APIs that take asynchronous jobs at 50% off, with results returned inside 24 hours.

Both also sell smaller mini and nano tiers for simpler tasks, and both keep lowering list prices through serving engineering. Continuous batching and quantization raise the tokens each GPU produces per hour, which lowers the cost of every token. On the open-weight side, the hosted model list covers models from z.ai, DeepSeek, Minimax, and others, and our GLM 5.2 benchmarks show how one of them performs on Telnyx.

Here is each lever side by side.

Cost levers on closed providers and on Telnyx

What the same levers cost on open-weight models

Telnyx runs open-weight models as inference as a service on GPUs it owns, priced up to 75% less than proprietary alternatives. Because Telnyx owns the hardware, no GPU rental margin sits on top of your rate. Cached input runs at up to 88% off, so the caching discipline you built for OpenAI carries straight over.

Three service tiers replace the batch and priority products you would otherwise buy separately. Default is standard capacity at standard rates, and every call uses it unless you set a tier. Priority selects faster serving capacity for low-latency interactive work, including voice orchestration. Flex gives you the lowest rates for background work where minutes of latency is fine, and every request stays synchronous. Flex is available in the US on select models. You switch tiers with one field in the request body, so there is no second provider to integrate.

Inference runs in-region across the Americas, Europe, MENA, and APAC, with zero data retention and no training on your data. Current per-token rates for each model and tier are on the inference pricing page.

Inference cost optimization: five steps that cut the bill

Cut your inference bill in this order: measure, route, cache, pick a tier, and switch. The last step moves your text traffic to open-weight models at up to 75% less, with a base URL change instead of a rewrite. Our inference cost optimization guide covers the engineering behind each step in more detail.

Which lever cuts my inference bill first?

• Long system prompt reused on every call: put the fixed prefix first so it caches. Cached input runs up to 88% off on Telnyx and up to 90% at closed providers.
• Overnight jobs nobody waits on: set the Flex tier in the request body on Telnyx Inference. Waiting work gets the lowest rates there, versus 50% off batch elsewhere.
• Agent that writes long answers: cap max output tokens and ask for terse replies. Output costs 3 to 5 times input, so verbose answers dominate the bill.
• Text workloads on a closed OpenAI model: point the base URL at Telnyx Inference and test an open-weight model. Savings reach 75% without rewriting your code.
• Spiky traffic, thinking of renting GPUs: stay on per-token billing with Telnyx Inference. A GPU idle half the hour doubles your cost per token, and Telnyx has no minimums.

Measure cost per feature before touching anything

Tag every request with the feature and the customer it serves, then work out tokens per user action and your cache hit rate. Prompt audits commonly find 30% to 50% of input tokens add nothing to output quality, and per-feature numbers show you where that waste sits. If you sell AI features to your own customers, as many software vendors do, per-customer tags also show which accounts cost more to serve than they pay.

Right-size the model and route cheap first

Send simple tasks such as classification, extraction, and short summaries to a smaller model, and escalate to a larger one only when confidence is low. A routed workload saves 60% to 90% on the traffic it routes. Run an eval before you send any traffic to a cheaper or quantized model, and rerun it on a schedule, because routing rules drift as prompts change. Read the AI gateway docs before you build a router of your own.

Structure prompts so the cache can hit

Put stable content first and variable content last. A cache hit needs an identical prefix, so your system prompt, tool schemas, and reference documents go at the top, and the user's message and session data go at the end. One timestamp near the top of the prompt breaks the prefix and drops every call back to the full input rate. Cap max tokens on every call, and turn reasoning off for tasks that do not need it, since hidden thinking bills at the output rate.

Pick the service tier that matches the latency you need

Match each workload to a tier by how long it can wait. The tier is one field in the request body, and it does not change the model or the context window, so moving traffic between tiers needs no new quality testing.

Which service tier should this traffic run on?

• Interactive chat where a user waits: Default.
• Voice agent orchestration or anything latency-sensitive: Priority.
• Offline evaluation, document processing, or background summarization where minutes are fine: Flex.
• A workload that needs a different model or context window: change the model, because the tier changes neither.

Move text traffic to open-weight with a base URL change

Telnyx Inference is OpenAI-compatible, so moving a feature means pointing your existing SDK at a new base URL and naming an open-weight model. The request below is the whole call: it sends one message to moonshotai/Kimi-K3 at api.telnyx.com/v2/ai/chat/completions.

For the same call in Python, see the Python reference example. If your code calls Claude through Anthropic's SDK, start with the Anthropic migration docs. Shortlist candidate models with our inference benchmarks, then run this check on each feature before you move it:

1. Pick 50 to 100 real production prompts for the feature.
2. Score the open-weight output against your current model on the same rubric.
3. Compare tokens per task on both, since a model that writes longer answers eats into the rate gap.
4. Route 10% of traffic first, and raise it as scores hold.

Self-hosting only beats an API at sustained high GPU utilization, so for most traffic the per-token API stays cheaper. If data residency is what pushes you toward your own hardware, read about edge inference first.

Inference cost FAQ

How much can we save by moving traffic from a closed model like Claude or GPT to an open-weight model?

Up to 75% on the traffic you move. That is the published gap between hosted open-weight models on Telnyx and proprietary alternatives, and cached input at up to 88% off applies on top. Your actual saving depends on how much of your traffic is text an open-weight model handles at the same quality, which your eval on real prompts will tell you.

Are open-weight models good enough to replace a closed model for production agents, and which traffic should move first?

Often, for text, and your eval decides it feature by feature. Move text work first: classification, extraction, summarization, and agent steps that call tools. Keep video and multimodal work on closed models for now, since open-weight models still trail there. Route 10% of a feature's traffic, compare scores against your current model, then expand.

Does cached-input pricing apply to hosted open-weight models, and how is it priced?

Yes. On Telnyx, cached input is billed at up to 88% off the standard input rate, and per-token rates vary by model and service tier. The prompt rule is the same as at OpenAI or Anthropic: keep your system prompt, tool schemas, and reference documents identical at the start of every request so the prefix can hit.

What is inference cost in AI?

It is what you pay each time a trained model processes a request and returns an output. Hosted APIs bill it per token, with separate rates for input, output, cached input, and batch or low-priority work. If you run your own GPUs, it is the GPU-hour price divided by the tokens that GPU produced in that hour.

When does self-hosting cost less than an inference provider?

When your GPUs stay busy around the clock. Cost per token is the GPU-hour price divided by the tokens produced, so every idle hour makes each token more expensive. Spiky or growing traffic rarely keeps a GPU saturated, which is why most teams pay less on a per-token API with no minimums until utilization stays high and steady.

Does quantization hurt model quality?

It can, and how much depends on the model, the precision, and the task. Lower precision raises tokens per GPU-hour and lowers cost, but some tasks lose accuracy. Treat a quantized model like any other model swap: run your eval on real prompts before you route traffic to it, and rerun it when the model or your prompts change.

Cut your inference bill by up to 75%

Move your text traffic to open-weight models hosted on Telnyx-owned GPUs and cut inference cost by up to 75%, with a base URL change instead of a rewrite. See the rates on the inference pricing page.

Try Telnyx Inference