Your dashboard says 300 ms, but users still wait. Here’s what time to first token really measures, how to time it yourself, and the fixes that bring it down.
Takeaways
- Users judge speed by when words first appear. Once streaming outpaces reading, quicker tokens go unnoticed.
- Self-measured numbers usually exceed what a host's metrics show, because your timer includes network time the server never sees.
- Reasoning models often think before they speak, so an early first token can still precede a long silence.
- TTFT shifts as inputs grow and servers get busier, so give short and long requests different goals.
- Trimming input, keeping prompts cache-friendly, and dialing back reasoning are the quickest fixes on your end.
What is time to first token (TTFT)?
Time to first token is how long you wait before an LLM starts answering. It's the number that decides whether a chat window feels instant or stuck.
You've probably lived this: the demo felt snappy on short test prompts, then real users pasted in long documents and replies started to crawl. For a refresher on what happens inside a model during a request, start with what AI inference is.
What fills the wait: network, queue, and prefill
Three things fill the time to first token: the network trip, the provider's queue, and prefill. TTFT, also called first token latency, ends when the first token lands, not the last, because LLM responses stream one token at a time.
- Network: your request travels from your code to the provider, through DNS, a TLS handshake, and usually a gateway or load balancer.
- Queue: if the server is busy, your request waits behind everyone else's.
- Prefill: the model reads your entire prompt and builds its KV cache (the stored attention values it reuses for every new token) before it can produce anything. Longer prompts mean longer prefill.
TTFT vs TPOT, ITL, and end-to-end latency
TTFT is one of four numbers that describe a streamed response, and each answers a different question.
| Metric | What it measures | What it decides |
|---|---|---|
| Time to first token (TTFT) | Request sent to first token received | How fast the reply seems to start |
| Time per output token (TPOT), also called inter-token latency (ITL) | Average gap between tokens after the first | How smoothly the reply streams |
| End-to-end latency | Request sent to last token received | When the full answer is ready |
| Tokens per second (TPS) | Output tokens per second, per user or across a server | Throughput and cost |
Definitions follow vLLM's metrics docs and NVIDIA's NIM benchmarking guide.
TPOT and ITL describe the same thing. Most tools calculate it as end-to-end latency minus TTFT, divided by output tokens minus one, though some fold the first token into the average, so check before you compare numbers from two tools.
TTFT gets the attention for a reason. The DistServe researchers point out that once tokens arrive faster than people read (about 250 words per minute), faster streaming stops mattering. The wait before the first word is the part users actually feel.
One caution: a provider can lead a TTFT chart and still finish answers last, so always check end-to-end latency alongside it.
How to measure time to first token
Your TTFT, your provider's dashboard, and a benchmark chart can all disagree about the same request, because each one starts or stops the clock somewhere different.
.svg)
Your clock vs the server's clock
Your clock and the server's clock measure TTFT from different starting points. Picture one streamed chat request with five timestamps:
- Your code sends the request.
- The server receives it and starts tokenizing.
- The first streamed chunk with any token arrives.
- The first chunk with answer text arrives.
- The last token arrives.
Your TTFT is 3 minus 1. The server's TTFT is 3 minus 2. Time to first answer token (TTFA) is 4 minus 1, and end-to-end latency is 5 minus 1.
vLLM, the most common open-source serving engine, starts its TTFT clock when its frontend receives the request and tokenization begins. Everything before that (DNS, TLS, the trip across the internet, a load balancer, an API gateway) sits in your number and nowhere on the server's dashboard.
That's how a dashboard can say 300 ms while your users complain. The gap isn't a measurement error. It's real time your user spends waiting.
Why the first chunk can be empty
The first streamed chunk is often empty because many APIs open with a chunk that carries only a role or metadata. NVIDIA's benchmarking tool skips responses with no content for exactly this reason: a TTFT stopped on an empty chunk measures nothing. Your timer should skip them too.
How to time TTFT and TTFA in code
To time TTFT and TTFA in code, run two clocks on the same stream: stop the first on any non-empty chunk, thinking included, and the second on the first chunk with real answer text.
This snippet works with any OpenAI-compatible endpoint. Providers name the reasoning field differently, so it checks the two common ones.
import os
import time
from openai import OpenAI
client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
start = time.perf_counter()
ttft = ttfa = None
stream = client.chat.completions.create(
model=os.environ["MODEL"],
messages=[{"role": "user", "content": "Explain what a SIP trunk is in two sentences."}],
stream=True,
)
for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
thinking = getattr(delta, "reasoning_content", None) or getattr(delta, "reasoning", None)
answer = delta.content
now = time.perf_counter()
if ttft is None and (thinking or answer):
ttft = now - start # first token of any kind
if ttfa is None and answer and answer.strip():
ttfa = now - start # first token of the answer
end = time.perf_counter()
def show(seconds):
return f"{seconds:.3f}s" if seconds is not None else "none"
print(f"TTFT {show(ttft)} TTFA {show(ttfa)} end-to-end {show(end - start)}")One run tells you very little. Repeat it and record p50 (the median) and p95 (the time 95% of requests beat), along with the prompt size and the region you ran from.
Time to first answer token: TTFT for reasoning models
With a reasoning model, a fast TTFT can hide a slow answer: the first token is usually the model thinking, and your user sees nothing until the answer starts.
In a benchmark Telnyx ran and published on GitHub, GLM 5.3 with reasoning set to high answered 230 requests across four providers. The first streamed piece was reasoning text in 153 of them, about two-thirds. With a 10,000-token prompt and a long answer, the median first token arrived after 1.18 seconds, but the first word of the answer took 9.18 seconds.
A TTFT chart would call that request fast. Treat these results as directional: one model, one or two days of runs, and ten requests per provider per prompt size, pooled across providers.
Time to first answer token is the fix. Artificial Analysis defines it as the wait until the first answer token, with reasoning time included, so reasoning and non-reasoning models get measured the same way. Report it next to TTFT for any reasoning model.
If users never see the thinking, TTFA is the wait they feel. If you show a "thinking" indicator, TTFT still matters, since it's the moment that indicator can appear.
Your biggest TTFA lever is reasoning effort: lower it, or turn reasoning off, for routine requests.
Why TTFT is a range, not a number
TTFT changes with prompt size and with how busy the server is, so a single TTFT figure tells you little until you know both.
How prompt size changes TTFT
Prompt size changes TTFT a lot, though not in a straight line. In the same Telnyx benchmark, short answers only, median TTFT was 0.44 seconds at about 900 prompt tokens, 1.00 second at about 9,500, and 4.77 seconds at about 89,000.
Ten times the prompt cost a little over double the wait at the small end and almost five times at the large end. On short prompts, fixed costs like the network and the queue dominate. On long ones, prefill takes over.
How load changes TTFT
Load roughly doubles TTFT, and long prompts pay the most in absolute terms. In a second Telnyx benchmark (GLM 5.3 Flash on two managed endpoints, October 2026), running ten requests at once instead of one moved median time to first answer like this:
- 1,000-token prompt: 0.34 to 0.55 seconds
- 10,000-token prompt: 0.72 to 1.47 seconds
- 100,000-token prompt: 4.16 to 8.76 seconds
Prefill is the work queued requests wait on, so the longer the prompt, the bigger the penalty.
.svg)
How to set a TTFT target
Set a TTFT target per prompt-size band, and judge it at the slow end, not the median. Users remember the worst waits, which is why benchmark bodies use p99 limits (the time 99 of 100 requests must beat).
MLPerf, the industry inference benchmark, shows how much the job matters. Its original Llama 2 70B chat test allowed 2 seconds of TTFT, its interactive chat test set a 450 ms p99 limit, and its long-context Llama 3.1 405B test allowed 6 seconds at p99.
One number for everything doesn't work either. Researchers behind the Etalon framework note that TTFT ignores prompt length, so a single fixed target isn't practical, and dividing TTFT by prompt length unfairly punishes short requests. A single TTFT number is close to useless until you know the prompt size.
How to reduce time to first token
Most TTFT fixes are yours to make: send less, send it closer, and ask for less thinking. The rest depends on how your provider runs its servers.
What you can change
You can reduce TTFT on your side with six changes:
- Send a shorter prompt. Trim retrieved context, old conversation turns, and tool definitions the model won't use.
- Make the prompt cacheable. A cached prefix is the unchanged tokens at the start of a prompt, so one changed character early on breaks it. Put fixed instructions first and per-user content last. Check your provider's minimum, too: OpenAI's guide sets a 1,024-token floor and notes caching doesn't guarantee lower latency on every request.
- Lower reasoning effort, or skip reasoning, for routine requests.
- Call a GPU in your users' region, so the network leg stays short.
- Stream the response, so users see the first token instead of waiting for the last.
- Reserve capacity if your traffic is steady, so requests don't queue behind other customers'.
What your provider can change
Your provider controls three more TTFT levers, and each is worth asking about:
- Prefix caching on the server: the SGLang authors show that reusing a cached prefix cuts prefill work, which lowers first-token latency.
- Chunked prefill: Sarathi-Serve splits a big prompt's prefill into chunks so it doesn't freeze streams already running. The DistServe authors found the trade-off lands on TTFT, since chunked prefill runs slower than full prefill.
- Separate prefill and decode GPUs: in its authors' tests, DistServe served up to 7.4 times more requests by splitting the two phases, and Microsoft's Splitwise reported up to 1.4 times higher throughput at 20% lower cost.
What doesn't reduce TTFT
Speculative decoding doesn't reduce TTFT, even though it often shows up on TTFT fix lists. It speeds up decode by drafting several tokens and checking them in parallel. The first token still waits on prefill, so the gain shows up mostly in time per output token.
Time to first token in voice AI agents
On a phone call, slow TTFT means silence, because the caller hears nothing until text-to-speech has words to say.
In voice AI agents, the LLM's TTFT is one stage of every turn. Caller audio arrives over a programmable voice API, speech-to-text transcribes it, the model starts answering, and text-to-speech speaks. With a reasoning model, the caller waits for TTFA, so hidden thinking turns into dead air.
Telnyx keeps every stage of that turn on one network. Telnyx's voice AI agent platform runs speech-to-text, the language model, and text-to-speech on GPUs in the same data centers as Telnyx's telephony core. Telnyx's Inference API serves open-source models on GPUs Telnyx owns, through OpenAI-compatible endpoints, and Telnyx publishes its inference benchmarks request by request, with time to first token and time to first answer token reported separately.
Frequently asked questions about time to first token
Quick answers to common questions about time to first token.
What does TTFT stand for?
TTFT stands for time to first token. It's the delay between sending a request to a large language model and receiving the first token of its reply, and it's also called first token latency. Because responses stream, TTFT decides how quickly a reply appears to start, even when the full answer takes much longer to finish.
What is the difference between TTFT and TPOT?
TTFT is the wait before the first token arrives. TPOT, or time per output token, is the average gap between the tokens that follow. TTFT decides how fast a reply seems to start, while TPOT decides how smoothly it streams. Once TPOT beats human reading speed, users stop noticing improvements, so TTFT usually shapes how fast an app feels.
How is time to first token calculated?
Time to first token is the timestamp of the first non-empty token minus the timestamp when the request was sent. A client-side clock starts when your code sends the request, while a server-side clock starts when the server begins tokenizing, so client numbers run higher. For reasoning models, also calculate time to first answer token, since the first token is often hidden thinking.
What is a good time to first token?
A good time to first token depends on the job and the prompt size. MLPerf's benchmarks offer reference points: a 450 ms p99 limit for interactive chat, 2 seconds for its original chat test, and 6 seconds at p99 for long-context work. Set separate targets per prompt-size band rather than one number for every request.
Is time to first token the same as latency?
No. Time to first token is one kind of latency: the wait until a response starts. End-to-end latency adds the time to generate the rest of the answer, so it's always longer. A model can post a fast TTFT and still finish slowly, so track both numbers when you compare providers or tune your own setup.
If you're building real-time voice or chat and need TTFT to stay low under real traffic, our team can help you test it. Contact Us
