Inference as a service: how it works and what it costs

What is inference as a service

Inference as a service is a cloud service where you send input to a trained model over an API and get the model's output back, billed on usage. The provider owns the GPUs, drivers, and serving software, so your team writes requests instead of running hardware.

Training produces a model by adjusting its weights on large datasets, usually over days or weeks on large GPU clusters. Inference is everything after that: each time your product asks the model a question, the model runs once and returns an answer. Training happens occasionally. Inference happens on every request, which is why it becomes the line on your bill that grows with your traffic.

Inference as a service, managed inference, and serverless inference

These names describe the same thing. Managed inference, serverless inference, model API, and inference as a service all mean a provider hosts the model and you call it over HTTP. Some pages shorten the term to IAAS, which collides with Infrastructure as a Service, the long-standing name for renting virtual machines.The distinction that matters is between a model endpoint you call and infrastructure you run yourself.

What an inference provider does

Like other cloud services, inference is metered: you pay for what you consume. In practice, a provider does four jobs for you:

• Hosts the model weights on its GPUs
• Exposes an HTTP endpoint your application calls
• Scales GPU capacity up and down as requests arrive
• Meters usage and bills you per token

Telnyx Inference is one example: a curated library of open-weight models hosted on GPUs Telnyx owns, behind an OpenAI-compatible endpoint, with per-token billing and zero data retention.

How inference as a service works

With serverless inference, the provider already hosts the model, so you never upload weights. You send prompts, and the provider handles everything between the request and the response.

What happens to one request

1. Your app sends a POST request with a model name and a list of messages to an OpenAI-compatible endpoint.
2. The provider routes the request to GPU capacity, ideally in your user's region.
3. The serving software tokenizes the input, splitting text into the units the model reads.
4. The model generates output tokens, which stream back to your app as they're produced.
5. The provider counts input and output tokens and meters the request.

Here's the documented request to the Telnyx chat completions endpoint. Swap in your own API key and run it from a terminal:

The shape of that request matches what you already send to a proprietary model API: a bearer token, a model name, and a messages array.

Default, Priority, and Flex service tiers

Most providers sell one lane per model. On Telnyx, a single field in the request body, service_tier, selects the serving capacity, so you don't need a second provider to get a faster or cheaper lane. The model stays the same across tiers.

• Default: standard serving capacity at standard rates. Every call uses it when you omit service_tier.
• Priority: faster serving capacity for low-latency interactive workloads, including voice orchestration. Available on select models.
• Flex: the lowest rates, for background work where minutes of end-to-end latency is acceptable, such as offline evaluation, document processing, and summarization. Requests stay synchronous. Available in the US on select models.

Check which tiers a model supports in the service tiers docs before you set the field.

Inference as a service vs self-hosted

The decision actually has three options, and each one bills you differently.

Should you call an inference API or self-host?

• Spiky traffic, stock open-weight model: call Telnyx Inference and pay per token. Idle hours cost nothing, while a GPU rented by the hour bills whether or not requests arrive.
• Already calling OpenAI, cost is creeping up: move to the Telnyx Inference OpenAI-compatible endpoint. Your existing request code keeps working once you swap the base URL and API key.
• Sensitive prompts, users in one region: run open-weight models in-region on Telnyx Inference. Zero data retention means prompts are not kept, and nearby GPUs cut round-trip latency.
• Own fine-tuned weights, steady GPU load: self-host on GPUs you manage. Serverless endpoints serve models the provider already hosts, and constant traffic keeps an hourly GPU fully used.

• Serverless API: the provider hosts a shared model and bills per token. Scaling is the provider's problem, and your choice is limited to the models on the menu.
• Dedicated endpoint: you reserve GPUs, usually billed per GPU-hour whether they're busy or idle. Performance is predictable, and you pay for the reservation.
• Self-hosted: you buy or rent hardware and run the drivers, a serving engine such as vLLM, security patches, and on-call yourself. You control everything, and you own every outage.

Managed inference trades control for convenience. You get less say over the hardware and model catalog, and in exchange you skip capacity planning. Deployments that scale to zero save money when traffic is idle, but they bring cold starts: Google Cloud describes a Cloud Run GPU service as ready for inference in under 30 seconds. In a chat window, that's a slow first answer. On a phone call, it's a caller who hangs up. Telnyx serves inference from an owned GPU pool that autoscales with no cold starts.

When self-hosting still wins

Self-hosting earns its overhead in three cases:

• Custom weights no provider hosts. If you need a model tuned on your own data, start with the Telnyx fine-tuning API before you rent hardware.
• On-premises requirements. Some regulations require the model to run on hardware you control.
• Steady, very high volume on one model. Owned GPUs pay off only when they stay busy around the clock. See what a B300 GPU rental actually costs once idle hours are counted.

Benefits of inference as a service

These are the published figures to check your current bill and latency against.

Cost: Open-weight models served per token on Telnyx cost up to 75% less than proprietary alternatives. You pay only for the tokens you use, so a quiet month costs less than a busy one.

Scale: Autoscaling is on by default, from 0 to 1000s of requests per second, with no capacity to plan around. Independent runs such as the GLM-5.2 inference benchmarks show how that capacity performs on real prompt profiles.

Flexibility: Because the endpoint is OpenAI-compatible, your existing SDK works. Switching models means changing a model name, and switching providers means changing a base URL, so you can test an open-weight model against your current one without rewriting code.

Privacy: Inference runs in-region, with zero data retention and no training on your data. Where the GPU sits decides where your prompts travel, which matters as soon as you serve users under regional data rules.

What drives AI inference as a service latency and performance

Latency comes from three places: the network distance between your user and the GPU, the hops between providers, and the wait for capacity on a shared GPU. The model's own speed sits on top of those.

The LLM metrics that matter: time to first token, tokens per second, and p95

• Time to first token (TTFT): the wait before the first word arrives. A bad number means a blank chat box or a caller hearing silence.
• Tokens per second: how fast output streams once it starts. A low number makes long answers crawl.
• p95 and p99 latency: the slowest 5% and 1% of requests. Averages hide them, and your least happy users live in them.
• Streaming support: whether tokens arrive as they're generated. Without it, a chat or voice interface waits for the full answer before showing anything.

For a deeper breakdown of each metric, see inference latency explained.

Where the GPU sits decides your latency floor

Every request pays for the distance between your user and the GPU, and no model optimization removes it. A voice agent pays more: when speech-to-text, the language model, and text-to-speech come from three providers, each handoff adds a network round trip. Then the request waits in line for capacity on shared GPUs, and a scale-to-zero deployment can add a cold start on top.

Telnyx addresses each of those. Its GPUs sit next to the network that carries your users' traffic, with inference in-region across the Americas, Europe, MENA, and APAC, so the distance shrinks. Speech, inference, and telephony run on one network, so a voice request crosses no provider boundaries and no borders. For interactive and voice workloads, the Priority tier selects faster serving capacity, and the owned GPU pool has no cold starts. For measured numbers, see the GLM-5.3 latency benchmarks and how Telnyx places GPUs on the network.

AI inference as a service pricing: how per-token billing works

With per-token pricing, you pay separately for the tokens you send and the tokens the model returns. Input tokens are your prompt, system instructions, and any retrieved context. Output tokens are the model's reply, and they usually cost more per token than input. Rates vary by model and by service tier, so the same request costs less on Flex than on Priority, and a small model costs less than a frontier one. On Telnyx, the cheapest frontier model, MiniMax-M2.7, starts at $0.30 per million tokens.

Cached input changes the math for most products. When the start of your prompt repeats across requests, such as a long system prompt or a fixed set of tool definitions, those tokens bill at a discount: up to 88% off on Telnyx. There are no GPU rental fees, no compute surcharges, and no minimums, so the bill tracks the work your product actually does. Idle hours cost nothing, which is the main structural difference from renting hardware.

Per-token, per-GPU-hour, and per-request billing compared

The billing model you choose should match the shape of your traffic.

How the three inference billing models compare

Per-GPU-hour pricing rewards traffic that keeps the GPU busy around the clock. Most LLM products don't look like that: traffic peaks during business hours and drops overnight, and a reserved GPU bills for every one of those quiet hours.

How to read an inference price list

1. Find your model on the inference API pricing page.
2. Note the input, cached input, and output rates per million tokens.
3. Confirm the service tier you plan to use, since rates differ by tier.
4. Estimate what share of your input repeats across requests, and price that share at the cached rate.
5. Multiply tokens per request by requests per day, then by 30.

The formula is: (fresh input tokens × input rate) + (cached input tokens × cached rate) + (output tokens × output rate).

For example, GLM-5.2 launched on Telnyx at $1.40 per million input tokens, $0.26 per million cached input tokens, and $4.40 per million output tokens. A month with 200 million fresh input tokens, 800 million cached input tokens, and 50 million output tokens costs $280 + $208 + $220 = $708. Check the pricing page for current rates before you budget.

Inference as a service use cases

Each workload asks something different of the service:

• Chat and support agents: streaming and a low time to first token. Default or Priority tier.
• AI agents: many model calls per task, so per-token cost compounds. Cached input on repeated instructions and tool definitions cuts the bill.
• Voice agents: speech-to-text, LLM, and text-to-speech in one chain. Priority tier on in-region GPUs.
• Batch document processing: throughput over latency. Flex tier.
• Classic ML scoring, such as fraud checks: fast, predictable, machine-readable answers. Structured output with JSON mode and regex constraints.

AI agents, voice pipelines, and batch processing

Agents multiply calls. One user task can trigger a dozen model requests as the agent plans, calls tools, and checks its results, so a price that looks small per request becomes the bill. Long-context open-weight models such as GLM-5.2 on Telnyx are built for that kind of multi-step tool use.

Voice pipelines multiply hops. Every conversational turn runs speech-to-text, the model, and text-to-speech in sequence, and a caller notices each pause. That's why co-located voice AI infrastructure matters more for voice than for chat. Teams running contact center AI combine all three patterns on a single call: an agent, a voice pipeline, and after-call summarization.

Batch work tolerates minutes of latency, so the Flex tier fits overnight document processing, offline evaluation, and bulk classification. In financial services, fraud scoring and transaction checks pair structured output with in-region processing.

What companies provide inference as a service

Providers fall into four groups:

• Hyperscalers: AWS, Microsoft Azure, and Google Cloud offer managed model endpoints. Google Cloud also offers GPUs on Cloud Run for teams that want to serve their own models.
• GPU clouds and developer platforms: DigitalOcean offers serverless inference alongside GPU instances, and CoreWeave offers managed endpoints on its GPU cloud.
• Open-model serverless providers: Together and Fireworks serve open-weight models per token.
• Telnyx: serverless open-weight models on GPUs it owns, in-region across the Americas, Europe, MENA, and APAC, with zero data retention.

Serverless in-region is the rare option

The differences between providers show up outside the US. Many offer in-region dedicated deployments, which means reserving GPUs, but few run serverless, pay-per-token inference in Europe or APAC.

Since then, the Dubai GPU launch brought serverless inference to MENA, and LATAM is next. A residency setting on a provider that routes across borders is a policy. A GPU in your users' region is a physical fact.

What to look for in an inference as a service provider

Use this checklist when you compare providers:

• Where inference physically runs, and whether prompts are retained or used for training
• Whether the endpoint is OpenAI-compatible, so switching is a base URL change
• Whether you can choose a latency or cost tier without adding a second provider
• Structured output support, such as JSON mode and regex constraints
• Who owns the GPUs, so upstream cloud outages and price changes don't land on you
• Published per-token rates, cached input pricing, and no minimums

If you plan to add voice later, check whether the same network also runs speech and telephony.

Data residency, retention, and compliance

Regional data rules care about where processing happens, not just where data is stored. A provider that serves Europe from US GPUs sends every prompt across the Atlantic and back. Ask where the GPU sits for each region you serve, how long prompts and completions are kept, and whether they feed model training. Telnyx runs inference in-region across the Americas, Europe, MENA, and APAC, keeps zero data retention, and doesn't train on your data. For workloads that must stay closest to users, see how edge inference places compute near the source.

Portability: OpenAI-compatible endpoints and model switching

An OpenAI-compatible endpoint accepts the same chat completions request format as the OpenAI API: the same messages array, the same streaming behavior, the same SDKs. That makes your provider a configuration value rather than a code dependency. If a model improves, a price changes, or a region opens, you change a base URL and a model name and redeploy nothing else.

How to start with an AI inference as a service API

You need four things: an API key from the Telnyx Mission Control Portal, a model name from the pricing page, the chat completions endpoint, and, optionally, a service_tier value for Priority or Flex.

Send your first chat completion

The cURL request earlier in this article is the smallest working call. The response is a standard chat completion object: the model's reply in choices, plus token usage counts you can check against the pricing page.

Switch models without redeploying

If your app already uses the OpenAI SDK, point it at the Telnyx base URL and keep everything else. This Python example streams a response from GLM-5.2:

To switch models, change the model string. Store it in configuration or a feature flag instead of hard-coding it, and you can move traffic between models at runtime without shipping new code.

How Telnyx delivers inference as a service

Models: Telnyx hosts a curated library of open-weight models from labs including z.ai, DeepSeek, Moonshot, and MiniMax, updated as new models release.

Regions and privacy: Inference runs in-region across the Americas, Europe, MENA, and APAC on GPUs Telnyx owns, with zero data retention and no training on customer data. LATAM is coming next.

Tiers and scaling: One service_tier field selects Default, Priority, or Flex capacity. Autoscaling runs from 0 to 1000s of requests per second with no cold starts, and structured output supports JSON mode and regex constraints.

One network: Inference, text-to-speech, voice AI agents, and telephony share the same API key, the same network, and the same bill. Moving from text to voice adds no new vendor.

Inference as a service FAQ

Check your bill against the published rates

Sign up, change your base URL, and compare your first bill against the published per-token rates and service tiers.

Try Telnyx Inference