---
title: "Edge inference: where to run AI models for low latency"
canonical: https://telnyx.com/resources/edge-inference
---
# Edge inference: where to run AI models for low latency

[Back to blog](https://telnyx.com/resources)

**Inference**

# Edge inference: where to run AI models for low latency

Where your model runs sets how fast your agent can answer. Compare device, on-prem, network edge, and cloud inference for real-time AI.

By [Eli Mogul](https://telnyx.com/authors/eli-mogul)

Takeaways

* Edge inference runs a trained model at the nearest tier with enough compute for the job: the device, an on-prem server, a GPU at the network edge inside the carrier network, or a central cloud region. Picking the right tier per workload sets your latency, your cost, and where your data travels.
* For real-time agents, the network path decides latency as much as the model does. Moving a model closer shortens only the network leg of each call.
* A single request field can move an interactive call, such as one turn of a voice conversation, onto latency-optimized capacity and a background job onto cheaper capacity, with the same model.
* Few serverless inference providers serve models in-region outside the US, so check where the GPUs physically sit before you compare anything else.

## What is edge inference?

Edge inference is a trained model taking an input and producing an output at a location close to your users or your data source, instead of in a distant central cloud region. The term gets used loosely. Some guides mean on-device processing only, some split the world into near edge and far edge, and others use on-device, gateway, and fog. A clearer map has four tiers, ordered by distance from the user.

**Which tier should run my model?**

* **Voice agent answering callers live:** call Telnyx AI Inference on latency-optimized capacity at the network edge. GPUs inside the carrier network cut the network leg of every turn.
* **Small model inside a camera or sensor:** run it on the device itself. The model fits hardware you own and the raw data never leaves the sensor.
* **Users in Europe, MENA or APAC:** call Telnyx AI Inference in-region with zero data retention. Prompts stay local instead of crossing into another provider's cloud and back.
* **Background jobs nobody waits on:** send them to cheaper capacity with one request field on Telnyx AI Inference. Same model as your live calls, lower cost per job.
* **Training on large datasets:** keep it in a central cloud region. Long GPU jobs need the biggest fleets, and training does not run on every request.

Telnyx runs inference at the third of those tiers, the network edge, on GPUs colocated with the network that serves your users. With [Telnyx AI Inference](https://telnyx.com/products/inference), requests are served in-region across the Americas, Europe, MENA, and APAC with zero data retention, so a prompt does not cross into another provider's cloud and back before your agent can answer.

**Run open models one hop from users**

Telnyx AI Inference serves open-weight models on in-region GPUs through an OpenAI-compatible API, so you change the base URL and keep your SDK.

**Try Telnyx Inference**

### What 'edge' means: device, on-prem, network edge

Each tier is defined by where the hardware sits and who owns it.

* **Device edge:** the model runs on the sensor, camera, phone, or vehicle that produces the data, and you or your customer own the hardware.
* **On-prem edge:** a gateway or server on your site runs one model for several devices, and you own and operate it.
* **Network edge:** a GPU in a regional point of presence inside the carrier network that already carries your users' traffic, which the network operator runs and you call as a service.
* **Central cloud:** a large region, often far from most of your users, with the biggest GPU fleets, run by a cloud provider.

Device and on-prem tiers give up compute to gain proximity. Central cloud gives up proximity to gain compute. The network edge sits between them: close enough for interactive loops, and large enough to run full open-weight models that would never fit on a phone or a camera.

### Edge inference vs edge computing vs edge AI

The three terms describe different scopes. Edge computing is about where compute sits, close to users or data. Edge AI is any AI workload at that location, including analytics or on-site fine-tuning. Edge inference is narrower still: only the step where a trained model produces an output.

In practice, training usually stays in a central data center, where large datasets and long GPU jobs live. Inference is the part that moves outward, because it runs on every request and every request carries the cost of distance.

Match each call to the right capacity

Send voice turns to latency-optimized capacity and background jobs to cheaper capacity with one request field and the same model.

[_Try Telnyx Inference_](https://telnyx.com/products/inference)

## How edge inference works

Every inference call spends time in four places: the network path to the model, queueing for serving capacity, the model's own compute, and for LLMs the wait until the first token streams back. Moving the model closer shortens only the first of these, a point covered in more detail in [inference latency explained](https://telnyx.com/resources/inference-latency). That is why tier choice matters most for small, frequent, interactive calls and matters least for batch jobs that run for minutes anyway.

The lifecycle behind that call follows the same pattern on every tier. You train or pick a model centrally, shrink it for the target tier with quantization, pruning, or distillation if the hardware is small, deploy it, run inference, and send only results or summarized features upstream. Then you monitor accuracy and drift and push updates to every location that runs the model.

**Note:** Moving the model closer shortens the network leg only. Queueing, model compute, and time to first token stay wherever the model runs. Measure all four before you decide a tier is fast enough. 

### Where the model runs: four tiers and their trade-offs

Each tier fits a different kind of workload, and the network edge row is the one most edge guides leave out.

**Where each inference tier fits**

| Tier         | Where the model runs                                                                 | Fits when                                                                                       |
| ------------ | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| Device       | On the sensor, phone, or vehicle                                                     | The device must work offline, reflexes need an answer in the same device, and the model is tiny |
| On-prem      | A gateway or server on your site                                                     | Several devices share one model and data cannot leave the building                              |
| Network edge | A GPU in a regional point of presence, one hop from the user. No inter-provider hops | Interactive agents, in-region data residency, and models too big for a device                   |
| Cloud        | A central region                                                                     | The largest models, training, and batch work with minutes of slack                              |

At the network edge, Telnyx places [GPUs on the network](https://telnyx.com/resources/inference-gpu-network) that serves your users, so requests avoid both cross-border routing and handoffs to another provider. The product keeps zero data retention, does not train on your data, and autoscales on owned GPU infrastructure with no capacity planning and no cold starts.

### What role do CPUs play in edge inferencing?

CPUs run everything around the model. They handle orchestration, pre-processing such as decoding audio, resizing video frames, and tokenizing text, post-processing of model outputs, and small models such as classifiers or wake-word detectors. GPUs and NPUs take the matrix-heavy forward pass of larger models, which is where most compute time goes.

On device and on-prem tiers, you pick a runtime that fits your hardware. [ONNX Runtime](https://onnxruntime.ai/) runs one model format across many chip vendors, while [NVIDIA TensorRT](https://developer.nvidia.com/tensorrt) compiles models for peak speed on NVIDIA GPUs. At the network edge, the provider manages the runtime and you send requests.

### What network do you need for AI inference at the edge?

The network has to carry your data to the tier that holds the model and back, fast and predictably, every time. For real-time agents that means four things:

* Low and stable round-trip time to the model, because variance hurts a conversation as much as raw delay does.
* Enough uplink for the raw data you still send, such as audio streams or camera frames.
* A [private connection to AWS](https://developers.telnyx.com/docs/network/vxc/aws) or another cloud region you fall back to, so overflow traffic stays off the public internet.
* Media handled on the same network as inference for voice, because audio that gets transcoded or downsampled between carriers can arrive at your speech model with less detail than it expects.

When you need your own logic next to the model, [Edge Compute](https://telnyx.com/edge-compute) runs your code on the same Telnyx network, so pre-processing, routing rules, and tool calls do not add another hop.

### Calling an in-region model from your own code

If you already use an OpenAI-compatible SDK, the network edge tier is one HTTP request. This is the documented chat completions call:

Shell

1234567

curl \-i \-X POST "https://api.telnyx.com/v2/ai/chat/completions" \\ \-H "Authorization: Bearer $TELNYX\_API\_KEY" \\ \-H "Content-Type: application/json" \\ \-d '{ "model": "moonshotai/Kimi-K3", "messages": \[{"role": "user", "content": "Hello, World!"}\] }'

Without a `service_tier` field, the request runs on Default capacity. Adding `service_tier` and selecting the Priority tier moves the same request onto serving capacity optimized for low-latency interactive work. Priority keeps the same model and context window, so you change one field in the request body instead of adding a second provider.

## Edge inference vs cloud inference for AI agents

Edge inference wins for AI agents whenever the agent sits inside a conversation or a control loop, because a short, stable network path cuts both delay and its variance. Central cloud wins when you need the largest models or the work can wait. Telnyx serves inference in-region, so your data stays where your users are, and its Priority tier exists for interactive workloads such as voice orchestration.

The trade-offs line up across five factors:

* **Latency and jitter:** edge tiers sit on a short path with little variance, while a distant cloud region sits at the end of an internet path that varies from call to call.
* **Model size:** device and on-prem tiers are bounded by local hardware, while network edge GPUs and central cloud run full open-weight models.
* **Data residency:** in-region inference with zero data retention keeps data in the user's region, while a central region may route it across borders.
* **Cost model:** Telnyx charges per token with no GPU rental fees or minimums, while other providers often bill per request or per GPU-hour.
* **Connectivity:** a device model works offline, the network edge needs the carrier path, and the cloud needs the public internet.

**Data residency is a customer question too:** in a Telnyx survey of 106 US adults (SurveyMonkey, 2026), 25% picked privacy and data concerns as their worry about interacting with a voice AI system (95% interval 18-35%). The figure is directional, but it shows the people on the other end of your agent care where their data goes.

### Latency requirements for real-time enterprise AI applications

An AI agent's latency requirement comes from the loop it sits in. A turn in a voice conversation has to come back before the caller notices silence. A robot's control loop needs an answer before the next actuation. A document pipeline can wait minutes without anyone noticing.

Those loops point to different tiers, and Telnyx exposes the same split as three serving tiers on the same models. Default is standard capacity. Priority is optimized for low-latency interactive work, including voice orchestration. Flex is for background work such as offline evaluation, document processing, and summarization, where minutes of end-to-end latency is acceptable.

### Where cloud-hosted inference adds latency and jitter

When a model is hosted by a third party in a public cloud, it sits at the end of an internet path you do not control, so you pay for both distance and variance. The symptom is familiar if you have chained a hosted speech-to-text vendor, a hosted LLM, and a hosted TTS engine: the agent sounds fine in a demo, then drags or talks over callers late in long calls, and no single vendor's dashboard shows why.

| "When you are actually using a inference or a TTS model that is hosted in a third-party vendor's servers, then they typically use a public cloud network. With that, that brings a lot of latency and jitter." **Abhishek Sharma, Senior Technical Writer at Telnyx** |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

Edge is faster per call, but it is not automatically faster end to end. A small model on a device that has to escalate a hard turn to a bigger model in the cloud pays for two inference calls and the trip between them, which can make the full response slower than one call to a capable model in-region.

### When a hybrid edge and cloud setup makes sense

Hybrid works best as a routing rule. A small model on the device or on-prem server makes the first decision, such as whether a frame shows a defect or whether a caller finished speaking, and a larger in-region model handles the turns that need reasoning. Batch summaries and evaluations then move to Flex capacity.

Because Telnyx tiers run on Telnyx-hosted models, moving a workload between Default, Priority, and Flex is one field change in the request rather than a second inference provider with its own contract and failure modes.

## Edge inference use cases where latency decides the outcome

The use cases that get the most from edge inference share one trait: someone or something is waiting on every answer. Telnyx runs text-to-speech, voice AI, and telephony on the same infrastructure as inference. Same API key, same network, same bill, so the whole loop of a voice agent can stay on one network.

### Real-time voice AI agents

A voice agent runs a full inference chain on every turn of the conversation.

| "We often obsess over LLM inference speeds, but in Voice AI, the network is often the silent killer." **Ian Reither, COO at Telnyx** |
| ------------------------------------------------------------------------------------------------------------------------------------ |

Audio arrives, speech-to-text produces text, the LLM writes a reply, and TTS turns it back into audio, while the caller waits through all of it. Every hop between vendors adds to the silence, as [voice agent latency compared](https://telnyx.com/resources/voice-ai-agents-compared-latency) shows across common stacks. Telnyx keeps the media plane, orchestration, and inference on one network, with Priority capacity for voice orchestration, and inference autoscales from 0 to 1000s of requests per second without capacity planning.

If you have a developer, the `edge-merge-ai-receptionist-python` reference example in the Telnyx code examples repository shows the inference step inside a live call flow, where an edge worker answers calls using Telnyx Voice, AI Inference, and Edge Compute. It is a reference example that has not been checked against the SDK, and its inference step calls the same chat completions endpoint shown earlier:

Python

1234567891011121314

def call\_inference(prompt, context): try: resp \= requests.post(INFERENCE\_URL, headers\=HEADERS, timeout\=15, json\={ "model": AI\_MODEL, "messages": \[ {"role": "system", "content": f"You are a professional receptionist. Use this context to route the call:\\n{json.dumps(context)}\\nBe concise. 1-2 sentences."}, {"role": "user", "content": prompt} \] }) if resp.ok: return resp.json()\["choices"\]\[0\]\["message"\]\["content"\] except Exception as e: app.logger.error("Inference failed: %s", e) return "Let me transfer you now."

| "Most 'end-to-end Voice AI' claims fall apart when you measure what actually matters: the end customer experience in real time. Turn latency, jitter, interruption handling. Not benchmarks on a slide, but behavior on a live call. That only happens when you control the full stack. Network, media plane, orchestration, and inference paths, not a fragile chain of third-party services over the public internet. This is the bar Voice AI should be held to." **Ian Reither, COO at Telnyx** |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

### Robotics, industrial inspection and predictive maintenance

Control loops and inspection lines need an answer before the next physical action. That points to the device or on-prem tier for the reflex, such as stopping an arm or rejecting a part, and to the network edge for heavier perception and planning models that will not fit on the machine. Teams that run fleets of robots place GPU capacity by region to hold a latency target across sites, and autonomous vehicles and smart-home devices follow the same split between on-board reflexes and remote reasoning.

### Regulated data in healthcare and finance

For patient and account data, the deciding factor is where the data may travel. In-region inference with zero data retention and no cross-border routing keeps a caller's details in the region where the caller is, which matters for [healthcare voice solutions](https://telnyx.com/solutions/healthcare) and for banks and insurers that answer to regional data rules.

**Voice AI agent**

Network edge. Every turn runs speech-to-text, an LLM, and TTS while the caller waits.

**Robot or inspection line**

Device or on-prem for reflexes, network edge for perception and planning models.

**Patient or account data**

Network edge in-region or on-prem, chosen by where the data is allowed to go.

## How to compare edge inference platforms

The first question for any edge inference platform is whether it serves models in the region your users are in. Kubernetes support, dashboards, and model catalogs matter less if every request still crosses an ocean. Telnyx runs serverless inference on its own GPUs in-region and prices it per token.

### A checklist for edge computing options for AI inference

Work through these seven criteria in order, starting with region:

* **In-region serving:** Telnyx serves inference in the Americas, Europe, MENA, and APAC.
* **Data retention:** zero data retention, and no training on your data.
* **API compatibility:** OpenAI-compatible endpoints, so you change the base URL and keep your SDK and orchestration.
* **Latency tier selection:** Default, Priority, and Flex, picked with the `service_tier` field.
* **Autoscaling:** owned GPUs scale with your traffic with no capacity planning.
* **Pricing structure:** per-token rates that vary by model and tier, up to 75% less than proprietary models, with no GPU rental fees or minimums.
* **Co-located runtime:** Edge Compute runs your own code next to the model.

### What vendors support edge inference at scale?

Many inference providers offer serverless APIs, but far fewer run those serverless APIs in-region outside the US. This is how Telnyx sees the market:

| "It's both, but the dirty secret is most competitors don't offer serverless in-region at all. Together is US-concentrated with no advertised EU or APAC serverless. Fireworks has 15-plus regions but dedicated deployments only outside the US. We run serverless inference in the Americas, Europe, and APAC today, with Dubai and São Paulo launching next. Data stays where your users are because our GPUs are physically there, not because you toggled a setting." **Regis David Souza, Senior Staff Engineer at Telnyx** |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

In practice, that means Together concentrates its serverless capacity in the US with no advertised EU or APAC serverless option, and Fireworks lists 15-plus regions but offers dedicated deployments only outside the US. If your users are in Frankfurt or Sydney, a dedicated deployment means paying for reserved GPUs, while a serverless in-region API lets you pay per token.

### Where Telnyx runs serverless inference today

Telnyx serves inference in-region across the Americas, Europe, MENA, and APAC, with LATAM coming soon. The GPUs sit inside the same network that carries your users' calls and messages, so data stays in the region where the request started.

Each model can run on three serving tiers. Default is standard capacity and is what every request gets when it omits the tier field. Priority is optimized for low-latency interactive workloads, including voice orchestration, and is live on select models today. Flex offers the lowest rates for throughput work where minutes of latency is acceptable, is available in the US on select models, and keeps all requests synchronous.

On price, Telnyx AI Inference costs up to 75% less than proprietary models, cached input is up to 88% off, and there are no GPU rental fees, compute surcharges, or minimums.

**FAQ**

**Can I use an inference API for just the LLM, speech-to-text or text-to-speech step and keep my own orchestration?**

Yes. Telnyx AI Inference exposes OpenAI-compatible endpoints, so you can swap only the LLM step by changing the base URL in your existing SDK. Your orchestration, prompts, and tools stay where they are. Speech-to-text and text-to-speech run on the same Telnyx network and API key if you later want to move those steps too.

**Can inference run in a specific region, such as the US West Coast and East Coast, to hold a latency target?**

Telnyx serves inference in-region across the Americas, Europe, MENA, and APAC, so requests from your users stay close to them. If you need capacity pinned to specific metro areas, such as both US coasts, talk to the Telnyx team about your latency target and traffic so they can confirm placement for your workload.

**How many concurrent inference requests can I run before I hit a limit?**

Telnyx AI Inference autoscales on owned GPU infrastructure, so you do not reserve capacity in advance and pay only for the tokens you use. For very high sustained volumes, or for limits on your own code running in Edge Compute, check the developer docs or ask the Telnyx team before launch.

**Is edge inference always faster than cloud inference?**

No. Edge inference shortens the network leg, but queueing, model compute, and time to first token stay the same wherever the model runs. A small edge model that has to escalate hard requests to a larger cloud model can be slower end to end than a single call to a capable model in-region.

**Can edge devices run large language models?**

Small ones, after quantization, on devices with enough memory. Larger open-weight models need data center GPUs. The network edge tier covers that gap: the model runs on GPUs in a regional point of presence near your users, so you get full-size models without the distance of a central cloud region.

**When does a hybrid edge and cloud inference architecture make sense?**

Hybrid makes sense when a quick local decision filters most traffic and only some requests need a bigger model. A camera can flag motion on the device and send only flagged frames onward, or a voice agent can detect end of speech locally and send the hard turns to an in-region LLM.

### Put inference one hop from your users

Run each call at the tier your latency budget needs, with Telnyx GPUs in-region across the Americas, Europe, MENA, and APAC. The API is OpenAI-compatible, so you change the base URL and you are live.

[Start building](https://telnyx.com/products/inference)

Share on Social

Eli Mogul

Content Writer & Editor

Eli is the content writer and editor at Telnyx. Born and raised in Chicago, Eli attended the University of Missouri where he obtained a BA in Journalism. Eli joined Telnyx in August of 2025\. In his spare time, you'll find Eli reading, playing video games, or running.

## Jump to:

[What is edge inference?](#what-is-edge-inference)[How edge inference works](#how-edge-inference-works)[Edge inference vs cloud inference for AI agents](#edge-inference-vs-cloud-inference-for-ai-agents)[Edge inference use cases where latency decides the outcome](#edge-inference-use-cases-where-latency-decides-the-outcome)[How to compare edge inference platforms](#how-to-compare-edge-inference-platforms)

## Sign up for emails of our latest articles and news

[Start building](https://telnyx.com/sign-up)

Related articles

* [**Inference**Open Source vs Frontier Models: Which Should Your Agents Run On?![Eli Mogul](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/Eli_Mogul_1de9939dcb.png?width=96&format=webp)By Eli Mogul14, Oct 2026](https://telnyx.com/resources/os-vs-frontier)
* [**Inference**NVIDIA B300 pricing in 2026: per hour, per system and per token![Eli Mogul](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/Eli_Mogul_1de9939dcb.png?width=96&format=webp)By Eli Mogul8, Oct 2026](https://telnyx.com/resources/nvidia-b300-pricing)
* [**Inference**We tested 3 open-weight models for multi-turn agents![Sonam Gupta, PhD](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/sonam_gupta_phd_2ae3a78101.png?width=96&format=webp)By Sonam Gupta, PhD8, Oct 2026](https://telnyx.com/resources/choose-open-weight-model)
* [**SIP Trunking**SIP response codes: a quick reference to every SIP code and error![Serhii](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/_DSC8799-6.jpg?width=96&format=webp)By Serhii Omelchenko14, Oct 2026](https://telnyx.com/resources/sip-response-codes)
* [**STT API**Telnyx STT against ElevenLabs Scribe on real phone audio![Abhishek Sharma](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/Abishek_Sharma_8313622a2b.png?width=96&format=webp)By Abhishek Sharma13, Oct 2026](https://telnyx.com/resources/telnyx-stt-vs-elevenlabs-scribe-benchmark)
* [**Insights and Resources**B300 GPU rental price per hour in 2026: what you actually pay![Eli Mogul](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/Eli_Mogul_1de9939dcb.png?width=96&format=webp)By Eli Mogul12, Oct 2026](https://telnyx.com/resources/b300-gpu-rental)

### Ask AI

* [![GPT](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/chatgpt_icon-ea55e09d.jpeg?width=92&format=webp)](https://chat.openai.com/?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Claude](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/claude_icon-c7c8eef3.webp?width=92&format=webp)](https://claude.ai/new?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Perplexity](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/perplexity_icon-cd5b74f1.webp?width=92&format=webp)](https://www.perplexity.ai/search?q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Gemini](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/gemini_icon-4ff1cbf4.webp?width=92&format=webp)](https://www.google.com/search?udm=50&aep=11&q=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)
* [![Grok](https://d2efw2akbx8gaf.cloudfront.net/https://dotcom-payload-assets-prod.s3.us-east-1.amazonaws.com/cms-media/grok_icon-e588e5bd.webp?width=92&format=webp)](https://x.com/i/grok?text=What makes Telnyx's global network different, breakdown call quality for normal phone calls and voice AI calls.)