Running Repeatable LLM Latency Benchmarks with Telnyx stateful actors

Developers building AI applications care about more than model availability. They care about how quickly a model starts responding, how long the full answer takes, and whether those numbers stay predictable as prompts and outputs get larger.

That is where repeatable benchmarking gets useful. A single request can tell you whether an endpoint works. A benchmark run can show how an inference workflow behaves across prompt sizes, output lengths, streaming responses, and retries.

Telnyx stateful actors are long-running, state-aware compute units for workflows that need to remember progress between requests. For inference workloads, that means an actor can own a job, track each model call, store timing metadata, and expose a clean way to poll or export results.

In this run, we used stateful actors to orchestrate Telnyx-hosted MiniMax M3 and GLM 5.2 benchmark requests from Telnyx infrastructure.

Inference metrics we measured

We focused on two user-facing latency metrics:

• E2E latency: time from request sent to final streamed token received.
• TTFT: time from request sent to first streamed token received.

For both metrics, lower is better. E2E latency matters when your app needs the full response before moving forward. TTFT matters when your product streams partial output and needs to feel responsive immediately.

The benchmark used the same six prompt profiles from the earlier runs:

• 1k input / ~100 output
• 1k input / ~1k output
• 10k input / ~100 output
• 10k input / ~1k output
• 100k input / ~100 output
• 100k input / ~1k output

Each model ran 10 repeats per profile.

Telnyx stateful actor run vs prior Telnyx baseline

The clearest story is the Telnyx-to-Telnyx comparison: prior non-stateful benchmark run versus the latest stateful actor run.

The largest improvement showed up in TTFT. MiniMax M3 moved from 1.87s to 0.61s p50 TTFT, while GLM 5.2 moved from 1.12s to 0.76s. E2E latency also improved: MiniMax M3 was roughly flat-to-better, and GLM 5.2 improved materially.

How this stacks up against prior provider baselines

We also compared the latest Telnyx stateful actor results against the earlier provider benchmark baselines. This is useful context, but it should be read carefully: competitors were not rerun through Telnyx stateful actors in this latest pass.

Against those prior provider baselines, Telnyx stateful actor results were ahead on p50 E2E for both models:

TTFT was especially strong for MiniMax M3. For GLM 5.2, TTFT was more mixed: Telnyx was roughly tied with Baseten at p50, slightly behind Baseten at p95, and still competitive with the other provider baselines.

Why stateful actors fit this pattern

A latency benchmark is a useful example of a broader AI workflow pattern:

1. Start a job.
2. Keep state while many model requests run.
3. Stream responses so TTFT and E2E can be measured separately.
4. Store structured results.
5. Export the final output for analysis.

That same pattern shows up in production AI systems: batch enrichment, document processing, evaluation runs, voice-agent analysis, and any workflow where a request may outlive a single synchronous HTTP call.

With Telnyx stateful actors, the application logic can keep workflow state close to the execution layer instead of pushing everything into a local script or external job runner.

Where to start

Telnyx inference uses OpenAI-compatible request patterns, so developers can bring familiar SDKs and switch the base URL. Stateful actors add a durable execution surface for workflows that need progress tracking, retries, and result export.

Useful Inference Docs:

• Telnyx inference getting started
• Telnyx inference models
• Telnyx stateful actors

Notes on the benchmark