Long-Context Benchmarks: Who's Actually Fast at 128K Tokens?

Markdown body

Running LLMs at long context is fundamentally different from running them at 4K or 8K tokens. The bottleneck shifts, the latency profile changes, and some models that look fast in short-context benchmarks fall apart at scale.

This post reports benchmarks we ran across major open-source models at 128K tokens of input context. All tests were run on GeneralCompute's inference infrastructure. We measured three things: time to first token (TTFT), decode speed after prefill completes, and effective throughput for a full request.

## Why Long Context Changes the Bottleneck

Standard benchmarks test models with a few hundred or a few thousand input tokens. At that range, the prefill phase (processing the input) is fast enough that TTFT is dominated by scheduling overhead and the first decode steps.

At 128K tokens, prefill dominates. Attention computation scales quadratically with sequence length under standard implementations. Going from 2K to 128K tokens is a 64x increase in sequence length, meaning roughly 4,096x more raw attention work before algorithmic optimizations kick in. Flash Attention brings this closer to O(n * d) in practice through tiling and recomputation, but the underlying trend holds: longer input means substantially higher TTFT.

KV cache size grows just as significantly. For a 70B-class model with grouped query attention (GQA), a 128K context can require 4-16 GB of KV cache depending on head count and layer depth. This memory pressure affects decode speed too, because every generated token must attend back to all 128K cached key/value pairs on each step.

## Methodology

Configuration for all runs:

- Input: 128,000 tokens of text (concatenated documents from a legal and financial corpus)
- Output: 512 tokens
- Batch size: 1 (single request, to isolate latency from throughput effects)
- Three runs per model, median reported
- Models loaded at native weight precision unless noted

We measured:

1. **TTFT**: Time from request submission to first output token
2. **Decode speed**: Tokens per second during generation (tokens 1-512 of output)
3. **Effective throughput**: Total output tokens divided by total request time (TTFT plus decode time)

## Results

### Time to First Token at 128K Input

| Model | Parameters | TTFT (128K input) |
|---|---|---|
| Qwen2.5 7B | 7B | 2.1s |
| Llama 4 Scout | ~17B active (MoE) | 4.2s |
| Gemma 3 27B | 27B | 8.7s |
| DeepSeek V3 | ~37B active (MoE) | 9.8s |
| Qwen2.5 72B | 72B | 12.4s |
| Mistral Large | 123B | 14.1s |

Phi-4 has a 16K official context limit and was excluded from the 128K test.

Llama 4 Scout's result deserves a note. It's a mixture-of-experts model where only a fraction of parameters are active per token. This keeps the compute per token lower than its total parameter count would suggest, and at 128K input that advantage compounds: faster TTFT than dense 72B models despite Scout being capable of contexts up to 10M tokens.

DeepSeek V3 similarly benefits from MoE architecture. Its ~37B active parameters per token put it ahead of dense models at comparable quality levels on TTFT.

### Decode Speed After 128K Prefill

Decode speed is measured once prefill finishes and the model begins generating. At this point, the KV cache is fully loaded and the model produces one token per step.

| Model | Decode Speed (tok/s) |
|---|---|
| Qwen2.5 7B | 94 tok/s |
| Llama 4 Scout | 71 tok/s |
| Gemma 3 27B | 38 tok/s |
| DeepSeek V3 | 28 tok/s |
| Qwen2.5 72B | 19 tok/s |
| Mistral Large | 16 tok/s |

Decode speed at long context is lower than what you'd see at short context. Each decode step must load 128K cached key/value pairs from memory, and memory bandwidth is the bottleneck. This is where GQA and MQA (grouped and multi-query attention) help: fewer attention heads means a smaller KV cache and less bandwidth required per decode step.

### Effective Throughput for a Full Request

Effective throughput accounts for both TTFT and decode time. For a request generating 512 output tokens from 128K input:

| Model | Total Request Time | Effective Throughput |
|---|---|---|
| Qwen2.5 7B | 7.5s | 68 tok/s effective |
| Llama 4 Scout | 11.4s | 45 tok/s effective |
| Gemma 3 27B | 21.2s | 24 tok/s effective |
| DeepSeek V3 | 27.8s | 18 tok/s effective |
| Qwen2.5 72B | 38.6s | 13 tok/s effective |
| Mistral Large | 45.6s | 11 tok/s effective |

The spread is significant. Qwen2.5 7B completes the same request in one-sixth the time Mistral Large does. Whether that trade-off makes sense depends entirely on whether 7B quality is sufficient for your task.

## What Drives the Differences

**Model size is the dominant factor**, but not the complete explanation. Smaller models are faster across all three metrics. The more interesting signal is within size buckets.

**MoE architecture helps more at long context than at short context.** Llama 4 Scout and DeepSeek V3 both show better TTFT than dense models at similar quality levels. During prefill, MoE models activate fewer parameters per token, reducing total compute. At 128K tokens, this advantage multiplies across the full sequence.

**GQA head count affects decode more than prefill.** Models using aggressive GQA (such as Qwen2.5 72B with 8 KV heads versus 64 query heads) have substantially smaller KV caches at long context. This partially explains why Qwen 72B holds up better on decode than Mistral Large despite both being in the 70-123B parameter range.

**Flash Attention variants matter.** Models running on backends with Flash Attention 3 support see meaningfully better TTFT at long context. The I/O reduction from tiling becomes more valuable as sequence length grows, because the ratio of compute to memory access becomes more favorable.

**Prefill parallelization has limits.** For very long sequences, even with maximum tensor parallelism, the raw compute for attention grows faster than you can throw hardware at it. This is why TTFT at 128K is unlikely to feel instantaneous on current hardware with current architectures, regardless of how many GPUs or ASICs you use.

## What This Means in Practice

**For document summarization and analysis**, where you load a large document and ask for a short answer, TTFT is the key metric. Llama 4 Scout and Qwen2.5 7B both come in under 5 seconds at 128K input, which is usable for interactive applications. Dense 70B-class models are in the 12-15 second range -- acceptable for batch processing, but problematic for anything user-facing.

**For long-context generation**, where you're producing hundreds or thousands of tokens from a large context (report generation, code completion over a full repository), decode speed matters more than TTFT. Qwen2.5 7B at 94 tok/s is about 5x faster than Mistral Large at 16 tok/s. The question is whether 7B quality clears the bar for your specific task.

**For agentic workflows** that make multiple sequential long-context calls, TTFT compounds in ways that become painful fast. Ten sequential calls with 64K tokens of context each, at 7 seconds TTFT per call, adds 70 seconds of wait time just from prefill. This is one reason agentic systems often benefit from keeping context lean through retrieval and summarization rather than accumulating a growing context window across steps.

**Batch throughput changes the picture.** These benchmarks cover single requests. At high batch sizes, the relative advantage of smaller models shrinks because larger models amortize fixed costs across more concurrent requests. If you're running 64 concurrent requests, the effective throughput gap between 7B and 72B narrows. The benchmarks above are most relevant for latency-sensitive, low-concurrency workloads.

## Choosing a Model for Long-Context Work

A practical decision framework:

- **TTFT under 5 seconds matters, 7B quality is sufficient**: Qwen2.5 7B or Llama 4 Scout are both good options.
- **70B-class quality needed, TTFT of 12-15 seconds is acceptable**: Qwen2.5 72B is the better choice between the dense options tested here.
- **70B quality with better TTFT than dense models**: DeepSeek V3's MoE architecture gives it a meaningful edge.
- **Context beyond 128K**: Llama 4 Scout supports up to 10M tokens and maintains reasonable performance throughout.
- **Batch-heavy workloads**: the TTFT difference matters less; focus on tokens-per-second at your actual batch size.

One thing these benchmarks make clear: 128K context is not yet fast enough for most real-time interactive applications, except with the smallest models. The fundamental bottleneck is prefill compute and KV cache bandwidth. Architectural changes like linear attention (Mamba, RWKV) and state space models promise O(1) per-token generation regardless of context length, but current versions haven't matched transformer quality at the frontier.

For most production use cases today, the practical answer is to keep context short where possible, use retrieval to fetch only the relevant chunks, and reserve full 128K context for tasks where there's no alternative.

## Running Long-Context Models on GeneralCompute

GeneralCompute supports Qwen2.5 7B, Qwen2.5 72B, Llama 4 Scout, DeepSeek V3, and others through an OpenAI-compatible API. Long-context requests use paged attention for memory efficiency, so you pay for the KV cache you actually use rather than pre-allocating for maximum context.

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

with open("large_document.txt") as f:
    document = f.read()

response = client.chat.completions.create(
    model="qwen2.5-72b-instruct",
    messages=[
        {
            "role": "user",
            "content": f"Summarize the key findings from this document:\n\n{document}"
        }
    ],
    max_tokens=512
)

print(response.choices[0].message.content)
```

If you want to benchmark your specific workload rather than relying on synthetic results, the GeneralCompute API lets you run the same request against multiple models and compare TTFT and decode speed directly. The numbers above give a baseline, but actual performance on your data and prompts may differ.

Check the [GeneralCompute docs](https://generalcompute.com) for current model availability and context limits.