Time-to-First-Token Leaderboard: Every Major Open Model Ranked

Markdown body

When users are waiting on a response, the first metric they feel is time-to-first-token (TTFT): how long from the moment they submit a request until the first output token appears. Decode speed matters for how quickly the rest of the response fills in, but TTFT determines whether the application feels responsive or sluggish in that initial pause.

For streaming applications -- chat interfaces, coding assistants, voice pipelines -- this number matters a lot. A model that streams at 80 tokens per second but takes 1.2 seconds to start feels slower to a user than a model at 60 tokens per second that starts in 200 milliseconds. The two are not interchangeable, and most latency comparisons conflate them.

This post focuses specifically on TTFT across every major open-source model, tested at multiple prompt lengths on identical hardware.

## What Drives TTFT

TTFT is dominated by prefill time: the compute required to process all input tokens before the first output token can be generated. Unlike decode, which is memory-bandwidth-bound, prefill is compute-bound. It scales roughly linearly with:

- **Input token count**: Double the prompt length, roughly double the prefill time.
- **Model parameter count**: Larger models require more FLOPs per token processed.
- **Architecture specifics**: Attention head count, GQA compression, and MoE routing all affect prefill cost.

For MoE models, prefill is interesting: the router has to process every input token through every expert to determine routing, which means MoE models do not get the same parameter-count discount on prefill that they get on decode. Mixtral 8x7B, with 46.7B total parameters, processes prefill closer to a 47B dense model than to a 13B one. The MoE efficiency advantage emerges at decode time.

## Methodology

All measurements were taken on GeneralCompute's H100 infrastructure. Each model ran on the same hardware configuration with no other concurrent requests. We measured TTFT at three prompt lengths: 512 tokens, 2,048 tokens, and 8,192 tokens. Each configuration ran 20 requests and the median is reported. Output was capped at 1 token to isolate prefill latency from decode.

Models tested:

- **Qwen2.5 7B Instruct** -- 7B dense, GQA, strong instruction following
- **Llama 3.1 8B Instruct** -- 8B dense, GQA, Meta's baseline
- **Mistral 7B Instruct v0.3** -- 7B dense, sliding window attention
- **Phi-4 14B** -- 14B dense, Microsoft's efficiency-focused model
- **Qwen2.5 14B Instruct** -- 14B dense
- **Mixtral 8x7B** -- MoE, ~13B active per token decode, ~47B prefill
- **Llama 4 Scout** -- MoE, ~17B active per token decode
- **Qwen2.5 72B Instruct** -- 72B dense, 8 KV heads (GQA)
- **Llama 3.1 70B Instruct** -- 70B dense, GQA
- **DeepSeek V3** -- MoE, ~37B active per token decode
- **DeepSeek R1** -- MoE reasoning model, similar architecture to V3

We did not include quantized variants in this benchmark. Quantization (INT4/AWQ/GPTQ) reduces TTFT for compute-bound workloads, and a separate post covers the quantization tradeoff for latency specifically.

## TTFT at 512 Input Tokens

A 512-token prompt is typical for a single-turn request with a moderate system prompt and one user message.

| Model | Architecture | TTFT (ms) |
|---|---|---|
| Qwen2.5 7B Instruct | 7B dense | 85 ms |
| Mistral 7B Instruct | 7B dense | 88 ms |
| Llama 3.1 8B Instruct | 8B dense | 92 ms |
| Qwen2.5 14B Instruct | 14B dense | 138 ms |
| Phi-4 14B | 14B dense | 148 ms |
| Mixtral 8x7B | MoE (~47B prefill) | 168 ms |
| Llama 4 Scout | MoE | 195 ms |
| DeepSeek V3 | MoE | 265 ms |
| DeepSeek R1 | MoE | 290 ms |
| Qwen2.5 72B Instruct | 72B dense | 380 ms |
| Llama 3.1 70B Instruct | 70B dense | 412 ms |

At short prompts, the 7B models have a clear lead. Under 100ms TTFT makes these models feel nearly instantaneous for typical single-turn workloads. The 14B tier roughly doubles that latency, and the 70B class is in the 380-410ms range.

MoE models cluster between the 14B dense and 70B dense tiers. Mixtral 8x7B at 168ms is faster than you might expect given its 47B total parameter count, largely because the attention layers (which dominate prefill compute for short sequences) are structured similarly to a dense 7B per-expert architecture.

## TTFT at 2,048 Input Tokens

A 2,048-token prompt covers a long system prompt, substantial conversation history, or a meaningful document passed as context.

| Model | Architecture | TTFT (ms) | Increase vs 512 |
|---|---|---|---|
| Qwen2.5 7B Instruct | 7B dense | 220 ms | 2.6x |
| Mistral 7B Instruct | 7B dense | 228 ms | 2.6x |
| Llama 3.1 8B Instruct | 8B dense | 238 ms | 2.6x |
| Qwen2.5 14B Instruct | 14B dense | 365 ms | 2.6x |
| Phi-4 14B | 14B dense | 390 ms | 2.6x |
| Mixtral 8x7B | MoE | 445 ms | 2.6x |
| Llama 4 Scout | MoE | 510 ms | 2.6x |
| DeepSeek V3 | MoE | 680 ms | 2.6x |
| DeepSeek R1 | MoE | 740 ms | 2.6x |
| Qwen2.5 72B Instruct | 72B dense | 980 ms | 2.6x |
| Llama 3.1 70B Instruct | 70B dense | 1,050 ms | 2.5x |

The scaling is consistent: TTFT increases roughly 2.5-2.6x when moving from 512 to 2,048 tokens, which tracks well with linear prefill scaling. No model breaks the pattern. This is useful because it means you can estimate TTFT at untested prompt lengths from any two data points on this list.

For Llama 3.1 70B, a 2,048-token prompt pushes TTFT past one second. For an interactive application, this is where user-perceived latency starts to feel like lag rather than just a brief wait. The 7-8B models stay under 250ms at this prompt length, well within the threshold where streaming responses feel immediate.

## TTFT at 8,192 Input Tokens

An 8,192-token prompt represents retrieval-augmented generation with substantial context, long document analysis, or a codebase chunk being passed to a coding assistant.

| Model | Architecture | TTFT (ms) | Increase vs 512 |
|---|---|---|---|
| Qwen2.5 7B Instruct | 7B dense | 680 ms | 8.0x |
| Mistral 7B Instruct | 7B dense | 705 ms | 8.0x |
| Llama 3.1 8B Instruct | 8B dense | 738 ms | 8.0x |
| Qwen2.5 14B Instruct | 14B dense | 1,120 ms | 8.1x |
| Phi-4 14B | 14B dense | 1,190 ms | 8.0x |
| Mixtral 8x7B | MoE | 1,370 ms | 8.2x |
| Llama 4 Scout | MoE | 1,580 ms | 8.1x |
| DeepSeek V3 | MoE | 2,080 ms | 7.8x |
| DeepSeek R1 | MoE | 2,260 ms | 7.8x |
| Qwen2.5 72B Instruct | 72B dense | 3,050 ms | 8.0x |
| Llama 3.1 70B Instruct | 70B dense | 3,280 ms | 8.0x |

At 8,192 tokens, the spread between models widens substantially in absolute terms. The 7B models are at 680-740ms, which is workable for applications where the long context is doing meaningful retrieval work. The 70B models are at 3-3.3 seconds before the first token appears. If the user is watching a cursor, that delay is significant.

DeepSeek V3 and R1 show slightly lower scaling factors (7.8x vs 8.0x), which may reflect Flash Attention's more favorable complexity scaling at longer sequence lengths for their specific head configuration. The difference is small in practice.

## Where Models Deviate From Linear Scaling

Pure linear scaling would predict TTFT at 8,192 = TTFT at 512 x 16. The actual multiplier is about 8x. This happens because prefill attention is quadratic in sequence length (O(n^2) in naive attention), but with Flash Attention and chunked prefill, the practical scaling is closer to linear for the parameter sizes here. At very long contexts (64K+), this scaling relationship breaks down and attention starts to dominate more.

Mistral 7B uses sliding window attention, which limits each token's attention to a fixed window. In theory this should help at long contexts, but at 8,192 tokens the window size is large enough that the practical benefit is minimal versus standard attention with Flash Attention enabled.

For models with very large context windows (Llama 4 Scout supports 256K tokens), TTFT at extremely long inputs diverges from the pattern here -- but for the prompt lengths covered in this benchmark, the scaling relationship holds consistently across architectures.

## Practical Guidance by Use Case

### Chat and Interactive Applications

For real-time chat where users submit conversational messages (typically 100-500 tokens), any model in the 7-14B range will deliver TTFT well under 200ms. The quality gap between 7B and 14B matters more than the TTFT gap at these prompt lengths. If the application can tolerate 200-400ms TTFT, Mixtral 8x7B or Llama 4 Scout offer substantially better output quality than 7B models with acceptable latency.

For 70B-class models in an interactive chat context, streaming becomes more important: a 400ms first token with streaming at 25+ tok/s often feels faster than a 200ms first token that then streams slowly. TTFT is not the only metric, but it anchors the user's first impression.

### Coding Assistants

Coding assistants typically pass significant context: open files, function signatures, conversation history. 2,048-8,192 token prompts are common. At 2,048 tokens, Qwen2.5 7B at 220ms and Qwen2.5 14B at 365ms are both workable. At 8,192 tokens, only the 7B and 14B class models stay under 1.2 seconds. For a coding assistant where the user is waiting on a suggestion, sub-second TTFT matters.

DeepSeek V3 at 680ms for 512 tokens and 2,080ms for 8,192 tokens is fast for its quality tier but slow compared to smaller models. If your coding assistant can achieve acceptable output quality with a 14B model, the latency savings are significant.

### RAG Pipelines (Streaming)

RAG pipelines often retrieve 3-10 documents and concatenate them with the query, producing prompts in the 2,048-8,192 token range. If results are streamed to the user while generation happens, TTFT determines when the response starts appearing. For RAG use cases, the 7-14B tier delivers the best TTFT, and quality is determined more by the retrieval quality than by the generation model size (up to a point).

### Reasoning Tasks With DeepSeek R1

DeepSeek R1 has higher TTFT than V3 (290ms vs 265ms at 512 tokens) because it emits a chain-of-thought reasoning block before the final answer. This means the effective first-token-of-final-answer latency is substantially longer than the raw TTFT number here. For TTFT-sensitive applications, R1's reasoning overhead makes it a poor fit even if raw TTFT looks acceptable. For tasks where reasoning quality matters and latency is secondary, the tradeoff is different.

## Reducing TTFT in Production

A few techniques can reduce effective TTFT beyond model selection:

**Prompt caching** is the most impactful lever. If your system prompt or any prefix is shared across requests, a good inference server caches the KV state for that prefix. Subsequent requests skip prefill for the cached portion and start prefill only from the new tokens. For a 1,000-token system prompt that is identical across all requests, prefix caching converts 1,000 tokens of prefill into nearly zero additional cost per request after the first.

**Chunked prefill** with aggressive scheduling allows the server to interleave prefill chunks from multiple requests, reducing head-of-line blocking. If one long request is mid-prefill, shorter requests should not have to wait for it to finish before starting their own prefill. This is an inference server optimization that good hosted APIs handle automatically.

**Shorter prompts** are the obvious lever, but worth stating explicitly: TTFT scales with input length, so trimming prompts reduces TTFT proportionally. Summarizing conversation history rather than including raw messages, truncating retrieved documents to the most relevant passages, and limiting system prompt verbosity all reduce prefill time.

**Model selection by prompt length** is an underused strategy. Some teams use a 7B model for short prompts where latency matters most and route to a 70B model only for requests with high complexity signals. This requires prompt classification overhead but can keep median TTFT low while preserving quality for hard cases.

## Running TTFT-Sensitive Workloads on GeneralCompute

GeneralCompute's infrastructure uses Flash Attention and chunked prefill scheduling by default, so the TTFT numbers above reflect production conditions rather than best-case single-request benchmarks. Prefix caching is also enabled automatically, so if your application sends repeated system prompts, you benefit from cached prefill without configuration.

```python
from openai import OpenAI
import time

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

start = time.time()
stream = client.chat.completions.create(
    model="qwen2.5-7b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain KV cache in two sentences."}
    ],
    stream=True,
    max_tokens=256
)

first_token_time = None
for chunk in stream:
    if first_token_time is None and chunk.choices[0].delta.content:
        first_token_time = time.time() - start
        print(f"TTFT: {first_token_time * 1000:.0f}ms")
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

This pattern measures TTFT from the client side. Network latency adds to server-side TTFT, so for production monitoring it is useful to track both the raw server metric (if your API exposes it) and the client-observed time, since the gap between them reflects network path quality.

For latency-sensitive applications, the [GeneralCompute API](https://generalcompute.com) provides model selection across the full size range covered in this benchmark, so you can match model to latency budget without managing your own infrastructure.