What Is Time to First Token (TTFT)? The LLM Latency Metric That Defines UX

Time to First Token (TTFT) is the elapsed time between sending a request to an LLM and receiving the first token of its response. It's measured in milliseconds, and it's the single metric that most directly shapes how responsive your AI application feels to users.

TTFT is distinct from total response latency. A model can have a high TTFT but still stream tokens quickly once it starts -- or it can start fast but generate slowly. Understanding the difference between these two behaviors, and knowing which one matters for your use case, is what this post covers.

What Is Time to First Token (TTFT)?

A Plain-English Definition

When you send a prompt to an LLM API, several things happen before you see any output: the server receives your request, loads or accesses the model weights, processes your entire input (the "prefill" phase), and then begins generating output tokens one at a time. TTFT is the wall-clock time from when your request hits the API to when the first output token arrives at your client.

A TTFT of 100ms means users wait a tenth of a second before anything appears. A TTFT of 800ms means they wait nearly a full second staring at a loading indicator.

Why TTFT Feels Different from Total Latency to End Users

Total latency (end-to-end) is the time from request to the last token. For a long response, this could be 10-30 seconds. But users don't perceive the full duration as waiting -- they perceive the gap before anything starts moving.

This is a well-studied phenomenon in human-computer interaction. Users tolerate long processes much better when feedback is immediate. An AI that starts responding in 100ms and takes 12 seconds total feels faster than one that takes 500ms to start and finishes in 8 seconds, even though the second option is objectively faster end-to-end.

The Psychology of Perceived Speed: Why Streaming Changes Everything

Before streaming APIs became standard, you had to wait for the complete response before displaying anything. The TTFT was the same as total latency from the user's perspective.

With streaming (using Server-Sent Events or WebSockets), the first token displays the moment it's generated, and subsequent tokens follow as fast as the model can produce them. This decouples perceived responsiveness from actual total latency. Users read at roughly 200-250 words per minute -- faster than most models generate tokens -- so a stream that starts quickly feels instant even if the full response takes several seconds.

For any real-time user-facing application, streaming isn't optional. And once you're streaming, TTFT becomes the most important latency metric to optimize.

The Four LLM Latency Metrics You Need to Know

TTFT -- Responsiveness

TTFT measures how fast the first token arrives. It's dominated by prefill time (processing your input) plus network round-trip time. Optimize this for conversational and interactive applications.

TPOT -- Time Per Output Token / Streaming Smoothness

Time Per Output Token (TPOT) is the average time between consecutive tokens during the decode phase. A TPOT of 25ms means the model generates 40 tokens per second. This determines whether streaming feels smooth or choppy -- if TPOT exceeds roughly 50ms (20 tokens/second), the output starts to feel slow to read.

Throughput -- Tokens Per Second Across All Users

System-level throughput measures how many tokens the inference server produces per second across all concurrent requests. High throughput is good for batch workloads and cost efficiency, but maximizing it often comes at the cost of higher per-request TTFT. See our tokens per second guide for a deeper look at this trade-off.

E2E Latency -- Full Request Round-Trip

End-to-end latency covers everything: network transit, queuing, prefill, decode, and the return trip. For short responses (1-3 sentences), E2E latency and TTFT are nearly the same. For long responses, they diverge significantly. Batch processing workloads often care only about E2E latency, not TTFT.

How These Four Metrics Relate to Each Other

E2E Latency = Network RTT + Queue Wait + Prefill Time + (TPOT × Output Tokens)
TTFT ≈ Network RTT + Queue Wait + Prefill Time

TTFT is effectively E2E latency minus the decode phase. This matters because the decode phase scales with output length, but TTFT does not. A 10-word response and a 1,000-word response have nearly the same TTFT -- assuming the same input prompt.

What Drives TTFT? A Technical Deep Dive

The Prefill Phase -- Why Longer Prompts Hurt TTFT

During prefill, the model processes every token in your input prompt in parallel to build the KV cache. This is computationally expensive: the compute cost scales roughly linearly with prompt length, and for long contexts it scales super-linearly due to attention complexity.

A system prompt with 2,000 tokens takes roughly 4x as long to prefill as one with 500 tokens. If your application has long system prompts or includes large documents in context, prefill time is likely your biggest TTFT bottleneck. Prefix caching can mitigate this by reusing KV cache states from repeated prefixes -- see the prefix caching post for how it works.

Hardware Bandwidth -- How Memory Speed Determines TTFT

Prefill is primarily compute-bound: GPUs and ASICs are doing matrix multiplications over the full input sequence. The memory bandwidth available determines how fast the model weights can be loaded into compute units for each operation.

This is why specialized inference hardware (ASICs and LPUs) can have significantly better TTFT than general-purpose GPUs. They're built with memory bandwidth and low-latency token generation in mind rather than general compute flexibility. The LPU vs GPU comparison covers these hardware trade-offs in detail.

Network Round-Trip Time -- The Non-ML Component

Even with perfect inference hardware, physical network latency adds to TTFT. A user in London hitting an inference server in Oregon adds 120-150ms of round-trip time before a single computation happens. For applications targeting sub-200ms TTFT, network proximity matters as much as model optimization.

Hosting your application server close to the inference provider's datacenter eliminates most of this overhead. Co-location within the same region can bring network RTT from 100ms+ down to under 5ms.

Batch Size and Queuing -- How Other Users Affect Your TTFT

Inference servers batch requests together to maximize GPU utilization. When a server is handling many concurrent requests, your request waits in queue before prefill starts. Under heavy load, this queuing delay can easily add 200-500ms to TTFT even if the underlying model is fast.

This is why TTFT benchmarks measured at idle (one request at a time) often differ substantially from production TTFT under load. A provider that advertises 80ms TTFT at idle might deliver 400ms TTFT when the cluster is at 80% utilization.

Model Size -- Smaller Models Are Not Always Faster Here

Counterintuitively, model size affects TTFT less than you might expect for short prompts. The prefill phase's compute cost scales with prompt length times model size -- so a 70B model processes a short prompt only slightly slower than a 7B model. For long prompts, the gap widens significantly.

The bigger effect of model size on TTFT comes from hardware availability: larger models require more GPU memory, which means fewer servers can run them, which means more queuing under load.

TTFT Benchmarks: Provider Comparison (2026)

Methodology -- How We Measured TTFT Fairly

All measurements use Llama 4 Scout (8B) with a 200-token system prompt and a 50-token user message. Requests were sent with a fixed concurrency of 10 (representing moderate production load) from a US-East server. TTFT is measured as time-to-first-chunk at the client. P50 and P95 are reported; we ran 500 requests per provider.

GeneralCompute vs Groq vs Together AI vs Fireworks

TTFT Under Load vs Idle -- The Real-World Gap

GPU-based providers show larger degradation under load because request queuing increases as the cluster fills up. Providers with custom hardware tend to maintain more consistent TTFT at higher utilization.

TTFT by Model Size -- 7B vs 70B vs 405B

Measured on GeneralCompute with a 200-token prompt, idle conditions:

The jump from 7B to 70B is roughly 2x. From 70B to 405B is another 2x. For latency-sensitive applications, choosing a smaller model when it's capable enough for the task is the highest-leverage optimization.

What Is a Good TTFT for Production?

Chatbots and Conversational AI -- Target Under 300ms

Human conversational turn-taking norms suggest that delays under 300ms feel natural, while delays above 700ms feel like a pause. For chat applications, TTFT under 300ms is the comfortable threshold, and under 150ms starts to feel near-instantaneous.

Voice AI -- Target Under 150ms (Non-Negotiable)

Voice applications have the tightest constraints. The full pipeline is: user speaks, ASR transcribes, LLM generates, TTS synthesizes, audio plays. Each stage adds latency. If the LLM contributes 300ms+ in TTFT alone, the total pipeline latency exceeds the 500-750ms window where voice responses still feel natural. Voice AI effectively requires TTFT under 150ms.

Coding Agents -- Target Under 200ms Per Step

Multi-step coding agents make many sequential LLM calls. If each step has 400ms TTFT and produces a 200-token response that takes 5 seconds to generate, a 10-step task takes roughly a minute. Halving TTFT from 400ms to 200ms saves nearly 20 seconds from that task -- a meaningful improvement for developer experience.

Batch Processing -- TTFT Is Irrelevant

For document processing, data extraction, or any non-interactive workload, TTFT doesn't matter. You care about throughput (tokens per second at scale) and cost. Optimizing for TTFT in batch workloads often means trading away throughput efficiency. Use different providers or configurations for batch vs interactive workloads if your application includes both.

How to Reduce TTFT in Your Application

Enable Streaming Responses (SSE / WebSockets)

If you're not streaming, users wait for the entire response before seeing anything. Enable streaming in the OpenAI-compatible API with stream: true:

import openai

client = openai.OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

stream = client.chat.completions.create(
    model="llama-4-scout",
    messages=[{"role": "user", "content": "Explain attention mechanisms"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

This alone is the single highest-impact change for perceived responsiveness.

Prompt Compression to Shrink Prefill

Shorter prompts prefill faster. Audit your system prompts for redundancy. A 3,000-token system prompt that could be 800 tokens will meaningfully improve TTFT. Tools like LLMLingua can compress prompts by 2-4x with minimal quality loss for many tasks.

For RAG applications, retrieve fewer but higher-quality chunks rather than stuffing the context window. Precision in retrieval reduces prefill time.

Choose an ASIC or LPU Provider for Prefill-Optimized Hardware

For applications where TTFT is the primary constraint, hardware architecture matters more than software optimization. Providers running custom inference ASICs or LPUs have hardware designed specifically for fast prefill and low first-token latency. This is often a 2-4x improvement over standard GPU providers for the same model.

Prefix Caching for Repeated System Prompts

If your application sends the same system prompt with every request (which most do), a provider with prefix caching can skip re-computing the KV cache for the shared prefix. This can effectively reduce the prefill cost to near zero for the repeated portion.

For a 2,000-token system prompt, prefix caching can cut TTFT in half.

Co-locate Your Application Server with the Inference Provider

Network RTT is a fixed floor on TTFT. If your application server is in us-east-1 and your inference provider's closest datacenter is us-west-2, you're adding 60-80ms of unavoidable latency before any inference starts. Deploying your application server in the same region as the inference provider removes this overhead.

TTFT vs Throughput: Which Metric Should You Optimize For?

The Fundamental Trade-off Explained

Inference servers can optimize for throughput or latency, but not both simultaneously. To maximize throughput, servers batch many requests together and process them in parallel -- this increases GPU utilization but adds queuing delay to each request, increasing TTFT. To minimize TTFT, servers process requests with low batching, which reduces utilization and raises cost-per-token.

This trade-off is inherent. No software optimization or hardware change eliminates it entirely, though specialized hardware narrows the gap.

Decision Framework -- When Each Metric Matters More

Optimize for TTFT when:

• Users are waiting for the response in real time
• The application is voice-based or agent-based with sequential steps
• Response time under 300ms is in your SLA

Optimize for throughput when:

• Processing documents, emails, or data in batch
• Running evaluations or fine-tune data generation
• Cost per token is more important than response time

Many production applications need both. The practical solution is using different model configurations or different providers for interactive vs batch workloads, rather than compromising both.

FAQ

What is a good TTFT for an AI chatbot?

Under 300ms P50 TTFT is the standard target for conversational chatbots. Under 150ms feels near-instant to most users. Above 600ms starts to feel slow relative to expectations set by tools like ChatGPT.

How is TTFT different from total latency?

TTFT measures only the time to the first token. Total latency (end-to-end) includes the full decode phase -- all the tokens after the first one. For short responses, these are similar. For long responses, total latency can be 10-30x higher than TTFT.

Does quantization affect TTFT?

Yes, but usually less than you'd expect. Quantization (INT4, FP8, etc.) primarily speeds up the decode phase, which runs memory-bandwidth-bound. Prefill is more compute-bound. Quantization typically reduces TTFT by 15-30% and reduces TPOT by 40-60%, so it helps more with streaming smoothness than with first-token latency.

Why is my TTFT slower in production than in benchmarks?

Three common causes: (1) your production traffic has longer system prompts than the benchmark used, increasing prefill time; (2) production has higher concurrent load, causing queuing delays; (3) your application server is geographically distant from the inference server. Benchmark TTFT numbers published by providers are almost always measured at low concurrency with short prompts -- these numbers are optimistic relative to real production workloads.

---

If TTFT is a constraint for your application, try GeneralCompute's API and run your own benchmark with your actual prompt lengths and concurrency levels. The numbers that matter are the ones measured under your real conditions.