---
title: What Is Machine Learning Inference? How It Works
canonical: https://telnyx.com/ai-inference/ml-inference
---
# What Is Machine Learning Inference? How It Works

[View as page](https://telnyx.com/ai-inference/ml-inference)

[Back to AI Inference](https://telnyx.com/ai-inference)

Last updated 17 Nov 2026

# Machine learning inference: how trained models make predictions in production

Machine learning inference is the moment a trained model starts working for you. See what it produces, how to serve it, and when you need a GPU.

By [Eli Mogul](https://telnyx.com/authors/eli-mogul)Content Writer & Editor

Machine learning inference is the moment a trained model starts working for you. See what it produces, how to serve it, and when you need a GPU.

**Takeaways**

* Training happens once in a while. Inference is the daily shift your model works for every request your product sends it.
* Most production ML is quiet work like scoring and ranking, and it has to finish before anyone notices a pause.
* Choose how to serve a model by asking who's waiting on the result, then size the hardware to match.
* You may not need a GPU at all. Plenty of smaller models run well on ordinary servers, especially after quantization.
* Judge a live model by its worst moments, not its average ones.

## What is machine learning inference?

Machine learning inference is running new data through a trained model to get an output. Training is the studying; inference is the model answering questions on the job, with its weights (the numbers it learned) locked in place.

### What is the primary function of the inference process in machine learning?

The primary function of the inference process in machine learning is to apply a trained model to new, unseen data to produce an output, such as a prediction, a classification, a score, or an embedding.

It's also where a model starts paying for itself. In a notebook, it helps no one; behind an endpoint, it checks every payment, email, or search your product handles.

You'll also hear it called inference in machine learning, ML inference, or model inference. The answer to "what is ML inference" fits in four steps:

1. You train the model once, and its weights are frozen.
2. Inference loads those weights into memory.
3. Each new input runs through a forward pass, one trip through the model's layers.
4. Nothing is learned along the way.

For the bigger picture, including training vs. inference, see [what is AI inference](https://telnyx.com/ai-inference/what-is-ai-inference).

### A machine learning inference example

Your inbox's spam filter is a machine learning inference example you used today:

1. An email arrives.
2. The text is split into tokens the model can read and converted into numbers.
3. The model runs a forward pass and returns a 0.97 probability of spam.
4. The mail app compares that score with its threshold, the cutoff above which it acts, and files the email as spam.

The model learned nothing from that email. It returned a number, and the app made the call.

## What machine learning inference produces

Machine learning inference returns whatever the model was trained to produce: a probability, a number, a vector, a ranked list, or a set of boxes. Your application decides what to do with it.

### Scores, labels, and predictions

Scores and labels are the most common outputs of machine learning inference. A classifier returns a probability, and your application turns it into a yes or no with a threshold: fraud or not, spam or not, about to cancel or not.

A regressor returns a number instead, like a likely sale price or next week's demand. Vision models return boxes and labels for what they see, and speech models return text.

### Embeddings and rankings

Embeddings and rankings power search results and "you might also like" rows. An embedding is a list of numbers that captures meaning, so a search for "cheap flights" lands near "budget airfare."

For search, inference turns your query into an embedding and a vector database finds the closest matches. For recommendations, a model scores candidates and ranking logic picks the few you see.

This is where much of the world's inference goes. Facebook reported that recommendation models made up [up to 79% of AI inference cycles](https://arxiv.org/pdf/1906.03109) in a production-scale data center. A later Meta paper put Facebook's AI power capacity at roughly 10:20:70 across experimentation, training, and inference, with much of that scoring running on CPUs.

Language models are a different serving problem: they write one token at a time, with a forward pass for each.

### Machine learning inference in action

Here's what inference looks like in six everyday places:

* At a bank, a card swipe sends the amount, merchant, and location to a fraud model. It returns a risk score before the terminal shows "Approved," and a high score triggers a text asking if the charge was you.
* In a contact center, a speech model transcribes the caller live and a second model tags the reason for the call, so the agent sees the right account early.
* On a factory line, a vision model checks a photo of each circuit board and flags missing or crooked parts for a human to inspect.
* In a food delivery app, a regression model turns distance, traffic, and kitchen load into the arrival time you watch tick down.
* At a streaming service, a batch job scores thousands of titles for every subscriber overnight, so your home screen is ready when you open the app.
* On a support team, embedding search finds past tickets about a "refund for late order" even when they never use the word refund.

The difference between them is who's waiting: the cardholder needs an answer in milliseconds, while nobody watches the streaming job at 3 a.m.

## Real-time, batch, and asynchronous inference

You serve a model in one of three ways, chosen by who is waiting for the answer: a person right now (real-time), nobody (batch), or a person who can wait a little while (asynchronous).

Real-time vs batch vs asynchronous inference

_Caption: Real-time, batch, and asynchronous inference differ in who is waiting for the answer._

### Real-time inference and its latency budget

Real-time inference answers one request at a time within a latency budget, the maximum time the answer is allowed to take. Tap "Pay," and a fraud model has milliseconds to score the charge.

Tight budgets leave hardware partly idle. In Google's 2017 TPU paper (Jouppi et al.), a 7 ms limit on 99th-percentile response time for one production model, MLP0, held a Haswell CPU to 42% and a K80 GPU to 37% of the throughput each reached with no limit.

Streaming is real-time inference that sends partial output as it's ready, which is why live captions appear as someone speaks.

On a phone call, you can hear every delay. The code below runs three inference calls in one back-and-forth: [text-to-speech](https://telnyx.com/products/text-to-speech-api) plays a phrase, [speech-to-text](https://telnyx.com/products/speech-to-text-api) transcribes the reply, and a language model scores it.

```py
import os
import requests

API = "https://api.telnyx.com/v2"
HEADERS = {"Authorization": f"Bearer {os.environ['TELNYX_API_KEY']}"}

def speak_phrase(text):
    # Text-to-speech: turn the target phrase into audio for the caller
    r = requests.post(
        f"{API}/text-to-speech/speech",
        headers=HEADERS,
        json={"text": text, "voice": "Telnyx.NaturalHD.astra", "output_type": "binary_output"},
    )
    return r.content  # audio bytes to play on the call

def transcribe_attempt(audio_bytes):
    # Speech-to-text: turn the caller's reply into text
    r = requests.post(
        f"{API}/ai/audio/transcriptions",
        headers=HEADERS,
        files={"file": ("attempt.wav", audio_bytes, "audio/wav")},
        data={"model": "openai/whisper-large-v3-turbo"},
    )
    return r.json()["text"]

def score_attempt(target, transcript):
    # Language model: score the attempt and suggest one fix
    r = requests.post(
        f"{API}/ai/chat/completions",
        headers=HEADERS,
        json={
            "model": "moonshotai/Kimi-K2.6",
            "messages": [
                {"role": "system", "content": "Score the attempt from 1 to 10 and give one tip."},
                {"role": "user", "content": f"Target: {target}\nHeard: {transcript}"},
            ],
        },
    )
    return r.json()["choices"][0]["message"]["content"]
```

The pause the caller hears is all three calls plus the network between them. To compare transcription speed by model, see the [STT models guide](https://developers.telnyx.com/docs/voice/stt/models).

### Batch and asynchronous inference

Batch inference scores a whole dataset on a schedule, like the overnight job that ranks a million customers by churn risk before the retention team logs in. You judge it on throughput and cost per item.

Asynchronous inference sits in between: you submit a request, get a ticket back, and collect the result later. It's a serving pattern rather than a formal term, and it suits long jobs like transcribing an hour-long meeting.

### How batching fits inside a latency budget

Batching groups requests that arrive within a few milliseconds of each other and runs them through the model together. Each request waits slightly longer, but throughput rises, as long as the group still finishes inside the latency budget.

In the Clipper serving system (Crankshaw et al., 2017), adaptive batching gave a scikit-learn linear SVM up to 26 times the throughput of no batching while staying within a 20 ms latency objective.

Make your first inference call

Create an API key in the Mission Control Portal and point your code at Telnyx.

[Sign up](https://telnyx.com/sign-up)[Contact us](https://telnyx.com/contact-us)

## CPU or GPU for machine learning inference

Many machine learning models run inference fine on a CPU. Deep learning inference with large inputs or heavy traffic is where a GPU pays off.

### When a CPU is enough

A CPU is enough for machine learning inference when the model is small, traffic is modest, or it runs on a device like a laptop or kiosk. Decision trees, linear models, and small neural networks usually qualify.

The gap is real for vision models. In [MLPerf Inference v5.1](https://docs.mlcommons.org/inference%5Fresults%5Fv5.1/), ResNet-50 took 7.63 ms per image on a CPU-only AMD Ryzen 9 7950X PC and 0.51 ms on a Lenovo ThinkEdge SE100 with one NVIDIA RTX 2000E Ada GPU (90th-percentile latency, edge SingleStream).

Quantization, storing weights in fewer bits, narrows it. A 2018 Google whitepaper by Raghuraman Krishnamoorthi found 8-bit weights make a model 4 times smaller and run 2 to 3 times faster than floating point on CPUs. For convolutional networks, post-training 8-bit quantization kept accuracy within 2%.

### When you need a GPU

You need a GPU for machine learning inference when you serve large deep learning models, high request volume, or large batches.

In the same MLPerf round, a dual Intel Xeon 6980P system served 18,704 queries per second on the DLRM recommendation model, while an 8x NVIDIA H200 system served 370,068, about 20 times more. MLPerf doesn't publish prices, so this says nothing about cost per query.

Here's how the two compare:

|                                                   | CPU                                                                | GPU                                                            |
| ------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------- |
| Good fit                                          | Trees, linear models, small networks, modest traffic, edge devices | Large deep learning models, high request volume, large batches |
| What helps                                        | INT8 quantization, batching                                        | Batching, mixed precision                                      |
| Edge example (ResNet-50, 90th-percentile latency) | 7.63 ms (AMD Ryzen 9 7950X PC, CPU only)                           | 0.51 ms (1x NVIDIA RTX 2000E Ada)                              |
| Datacenter example (DLRM, Server scenario)        | 18,704 queries/s (2x Intel Xeon 6980P)                             | 370,068 queries/s (8x NVIDIA H200)                             |

## Deploying a model for inference

Deploying a model for inference means turning a trained model file into something your application can call: an exported model, a runtime that executes it, and an endpoint in front.

### From trained model to endpoint

You export the model from its training framework, load it into an inference runtime, and put an HTTP endpoint in front so your app can send requests. Language models need more serving machinery, since they generate long outputs one token at a time.

Or you skip running the stack. A managed inference API handles hardware and runtime for you; self-hosting trades upkeep for control. The [Telnyx Inference quickstart](https://developers.telnyx.com/docs/inference/quickstart) shows a hosted model call in a few lines.

### What production adds around the model

In production, the model is the small part. In "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015), Google engineers showed model code is a small fraction of a real ML system; the rest is data collection, serving, monitoring, and configuration.

That's why a model that flew in your notebook can crawl behind an endpoint. Production adds versioning for rollbacks, monitoring for latency and errors, and a watch for drift, the shift of live data away from the training data. A fraud model trained on last year's habits will miss this year's scams.

Telnyx's [Inference API](https://telnyx.com/products/inference) serves open-weight models on GPU infrastructure Telnyx owns: language models through an OpenAI-compatible Chat Completions API, and [embedding models](https://developers.telnyx.com/docs/inference/embeddings) through an OpenAI-compatible embeddings endpoint. Telnyx runs inference GPUs in [five regions on four continents](https://developers.telnyx.com/docs/inference/models/regions), and calling a regional endpoint sends your request to the nearest GPU region.

## Measuring machine learning inference performance

Machine learning inference performance comes down to three numbers: how long one request takes, how many requests you can serve per second, and how slow the slowest requests are.

### Latency, throughput, and tail latency

Latency, or inference time, runs from the moment a request leaves to the moment the answer returns. It falls into three buckets:

* The forward pass, which is what most benchmarks report
* Engine overhead, such as queueing and waiting for a batch to fill
* The network path to the endpoint and back

Throughput is requests served per second. Tail latency is the response time of your slowest requests, usually reported as p95 or p99 (the time 95% or 99% of requests beat).

The tail is what people remember: one hung checkout in a hundred is one frustrated shopper per hundred. For deep learning inference behind a live product, watch p99, not just the average.

### What benchmarks measure for classic models

Public benchmarks have mostly moved on from classic models. By our count of MLPerf Inference closed-division results, classic models (ResNet, RetinaNet, 3D-UNet, BERT, DLRM, and YOLO) fell from 136 of 609 results in v5.1 (22%) to 59 of 504 in v6.0 and 18 of 476 in v6.1 (4%). The v6.1 closed division's 18 are all DLRMv3 and YOLO.

Recommendation still sets a clear speed bar: DLRMv2 had a 60 ms Server latency limit in v5.1, and DLRMv3 has 80 ms from v6.0 on. To read results against your own traffic, see [inference benchmarks](https://telnyx.com/ai-inference/inference-benchmarking).

### Frequently asked questions about machine learning inference

Quick answers to common questions about machine learning inference.

**Is inference the same as prediction in machine learning?**  
Not quite. Prediction is the word for a classifier's or regressor's output, like "this email is spam." Inference is the act of running any trained model. Every prediction is an inference, but embeddings and generated text are inferences that aren't predictions. In statistics, "inference" means drawing conclusions about a population from a sample (see [what is AI inference](https://telnyx.com/ai-inference/what-is-ai-inference)).

**Can machine learning inference run on a CPU?**  
Yes, for many models. Decision trees, linear models, small neural networks, and modest traffic all run well on CPUs, and quantizing weights to 8 bits makes inference 2 to 3 times faster than floating point. Large deep learning models, high request volume, and large batches are where a GPU earns its cost.

**Can you deploy your own trained or fine-tuned model on a managed inference service?**  
It depends on the provider. Many managed inference services host only the models in their catalog, while others let you bring your own, sometimes on dedicated hardware. Check the provider's docs for custom or dedicated deployments before you commit. For Telnyx, the [hosted model list](https://developers.telnyx.com/docs/inference/models) shows which models you can call today.

**Does a hosted inference service train on your data?**  
Policies differ by provider, so read the data processing terms before you send production data, including how inputs, outputs, and logs are handled. For Telnyx, customer data handling for model training is governed by Sections 3.1 and 3.5 of the Telnyx [Data Processing Addendum](https://telnyx.com/legal/data-processing-addendum).

**What is model inference?**  
Model inference and machine learning inference mean the same thing: running a trained model on new input to get an output. When your banking app sends a transaction to a fraud model and gets back a risk score of 0.12, that request is model inference. The model applies its frozen weights and learns nothing from your purchase.

---

Run open-weight models through an OpenAI-compatible API on GPU infrastructure Telnyx owns.

[**Try Telnyx Inference**](https://telnyx.com/products/inference)

Eli Mogul

Content Writer & Editor

Eli is the content writer and editor at Telnyx. Born and raised in Chicago, Eli attended the University of Missouri where he obtained a BA in Journalism. Eli joined Telnyx in August of 2025\. In his spare time, you'll find Eli reading, playing video games, or running.

[Start building](https://telnyx.com/sign-up)