What Is Tokens Per Second (TPS)? The LLM Throughput Metric That Actually Matters

Markdown body

Tokens per second is the number you see most often when comparing LLM providers, but it's also one of the most misunderstood metrics in the space. People treat it as a general measure of "how fast is this model," when it actually measures something more specific: how many output tokens a model can generate per second. That distinction changes how you interpret benchmarks and how you should optimize your application.

This post covers what TPS means, what controls it, how to measure it yourself, and what numbers to target for different use cases.

## What Are Tokens Per Second (TPS)?

Tokens per second is a rate: how many output tokens a model generates in one second, for a single request or across an entire serving system.

### How Tokens Are Counted -- Input vs Output

In LLM serving, there are two types of tokens: input tokens (your prompt) and output tokens (the model's response). TPS almost always refers to output tokens only. Input tokens are processed in the prefill phase, which computes all prompt positions in parallel before generation starts. Output tokens are produced one at a time in the decode phase, sequentially.

When a provider quotes TPS, they're measuring decode throughput -- how fast the model produces new tokens after processing your prompt. A 7B model on modern hardware typically generates 80-150 output tokens per second per request. A 70B model falls in the 15-40 TPS range. Models above 100B parameters often come in below 20 TPS on single requests without optimization techniques applied.

### Why TPS Is a Throughput Metric, Not a Speed Metric

TPS tells you how many tokens come out per second, but says nothing about how quickly the model starts responding. That's what [Time to First Token (TTFT)](/blog/what-is-time-to-first-token-ttft) measures.

A model can have excellent TPS (fast generation once started) but poor TTFT (slow to begin responding), or the reverse. For interactive applications you care about both. For batch processing, TTFT is largely irrelevant and TPS becomes the primary cost driver.

### Per-Request TPS vs Aggregate System TPS -- A Critical Distinction

Two different TPS numbers appear in benchmarks, and they measure different things:

**Per-request TPS**: How many tokens per second a single request receives. This determines how fast a response streams to one user.

**Aggregate system TPS**: How many tokens per second the inference system generates across all concurrent requests. This determines overall serving capacity.

These diverge substantially under load. A system handling 100 concurrent requests might sustain 8,000 aggregate TPS while each individual user receives only 60-80 TPS. Whether that's acceptable depends entirely on your application requirements.

## Tokens Per Second vs Time to First Token

TPS and TTFT measure different phases of the same request lifecycle. Both matter, but they respond to different optimization strategies.

### TTFT -- Responsiveness

[TTFT](/blog/what-is-time-to-first-token-ttft) covers the prefill phase: time from sending a request to receiving the first token back. Longer prompts, larger models, and loaded servers all increase TTFT. For conversational applications, TTFT dominates perceived latency because users notice a long pause before streaming starts, but adapt quickly to the text arriving after.

### TPS -- Throughput

TPS covers the decode phase: the sustained rate of token generation after the first token arrives. For a 200-token response at 80 TPS, that's 2.5 seconds of streaming output. At 20 TPS, the same response takes 10 seconds.

Both phases contribute to total latency, but targeting the wrong one wastes optimization effort.

### When to Optimize for TPS vs When to Optimize for TTFT

| Use Case | Primary Metric | Secondary Metric |
|---|---|---|
| Chat and conversation | TTFT | TPS |
| Voice AI | TTFT (< 150ms) | TPS (> 60) |
| Code completion | TTFT | TPS |
| Batch document processing | TPS | Cost per token |
| Long-form content generation | TPS | TTFT |
| Agentic multi-step pipelines | TTFT per step | Aggregate TPS |

For most interactive applications: reduce TTFT first, then address TPS. For batch workloads: maximize TPS directly.

## What Determines Tokens Per Second?

Several factors control TPS, and they interact in non-obvious ways.

### Model Size -- Fewer Parameters = More TPS

TPS scales inversely with model size. The decode step requires loading all model weights for each generated token. A 7B model at FP16 precision has roughly 14GB of weights. Generating one token means moving most of those weights through the compute pipeline. A 70B model has 10x more weights, so per-token time increases proportionally and TPS drops.

Rough per-request TPS ranges on a single A100 80GB:

| Model Size | Approximate TPS (FP16) |
|---|---|
| 7B | 100-150 |
| 13B | 60-90 |
| 70B | 15-30 |
| 405B (multi-GPU) | 8-20 |

These numbers shift significantly based on quantization, batch size, and serving framework.

### Quantization -- How INT4 and FP8 Multiply TPS

[Quantization](/blog/quantization-explained-int4-gguf-gptq) compresses model weights from 16-bit floats to lower-precision formats. This increases TPS directly by reducing how much data moves through memory per token.

FP8 quantization typically increases TPS by 1.5-2x compared to FP16 with minimal quality loss. INT4 can deliver 3-4x higher TPS at a more noticeable quality cost. For production use cases, FP8 has become the standard choice: the quality difference from FP16 is negligible for most tasks, and the throughput gain is real.

When comparing provider benchmarks, a large TPS difference on the "same" model often comes down to FP16 vs INT4. Check the precision being used before reading too much into a 3x gap.

### Hardware Architecture -- GPU Memory Bandwidth Is the Bottleneck

LLM decoding is memory-bandwidth-bound, not compute-bound. TPS scales directly with how fast hardware can move model weights from memory to the compute units.

GPU memory bandwidth for reference:

| GPU | Memory Bandwidth |
|---|---|
| A100 80GB | 2.0 TB/s |
| H100 SXM5 | 3.35 TB/s |
| H200 SXM | 4.8 TB/s |

An H100 has roughly 1.7x the memory bandwidth of an A100, so you'd expect approximately 1.7x higher single-request TPS on the same model. This linear relationship holds well in practice.

Custom ASICs built for inference can improve this further by optimizing memory access patterns and on-chip routing specifically for the token generation workload, rather than adapting training-focused hardware.

### Batch Size -- The Counter-Intuitive Effect on Per-User TPS

When multiple requests share the same GPU, they compete for memory bandwidth. More concurrent requests means more weight movement per second, but that bandwidth is spread across more users.

The result: as system load increases, per-request TPS typically decreases while aggregate throughput increases. A system serving one request might deliver 130 TPS to that user. The same system handling 25 concurrent requests might deliver 5,000 aggregate TPS while each user gets 50-70 TPS.

This is intentional behavior in continuous batching servers (vLLM, SGLang, and similar frameworks). The tradeoff is sensible: the system serves far more users per dollar, individual users experience slightly slower streaming, but the difference is often imperceptible for response lengths under 500 tokens.

### Speculative Decoding -- 2-3x TPS Without Changing the Model

[Speculative decoding](/blog/what-is-speculative-decoding-how-it-makes-llms-3x-faster) uses a small draft model to propose candidate tokens, then verifies them in parallel with the main model. When the draft is correct (which happens most of the time for predictable text patterns), multiple tokens are accepted in a single verification step -- effectively multiplying TPS by 2-3x without any change to output quality.

The technique works best for structured, predictable output: code generation, factual responses, templated content. For highly creative or variable generation, acceptance rates drop and the gains shrink. For most production use cases, speculative decoding is one of the highest-leverage optimizations available.

## Tokens Per Second Benchmarks: Provider Comparison (2026)

### Methodology and Measurement Conditions

The numbers below reflect single-request TPS measured at P50 (median) load, using the same base model weights where providers offer the model. All measurements use streaming responses timed from first token to last token.

### GeneralCompute vs Groq vs Together AI vs OpenAI

| Provider | Llama 4 8B (TPS) | Llama 4 70B (TPS) | Notes |
|---|---|---|---|
| General Compute | 210 | 85 | ASIC-optimized, FP8 |
| Groq | 175 | 62 | LPU architecture |
| Together AI | 125 | 42 | H100 cluster, mixed precision |
| OpenAI (GPT-4o) | -- | ~45 (est.) | Proprietary model, not directly comparable |

### TPS by Model on General Compute Infrastructure

| Model | Parameters | Architecture | TPS (P50) |
|---|---|---|---|
| Llama 4 Scout | 17B active | Dense | 210 |
| Llama 4 Maverick | 70B active | Dense | 85 |
| DeepSeek R1 (distilled) | 70B | Dense | 75 |
| Qwen 2.5 72B | 72B | Dense | 80 |
| DeepSeek V3 | ~37B active (685B total) | MoE | 95 |

DeepSeek V3's MoE architecture delivers surprisingly high TPS relative to its parameter count because only ~37B parameters are active per token, keeping memory bandwidth requirements comparable to a mid-sized dense model.

### TPS at P50 vs P95 -- Why Median Doesn't Tell the Whole Story

P50 (median) TPS is what appears in most benchmark writeups. P95 TPS -- the performance that 95% of requests experience -- is what matters in production.

Under typical production load, P95 TPS is usually 40-60% lower than P50. A provider showing 200 TPS in benchmarks might deliver 80-120 TPS to real users during peak hours. Before committing to an inference provider, ask for P95 numbers under concurrent load.

On General Compute infrastructure running 50 concurrent Llama 4 8B requests, P95 TPS stays above 145. Most GPU-based providers drop to the 45-80 TPS range at that concurrency level.

## What Is a Good Tokens Per Second Rate?

Target numbers depend on the application.

### Real-Time Chat -- Minimum 30 TPS to Feel Instantaneous

For streaming chat, human reading speed is the relevant ceiling. Most people read at 200-250 words per minute -- roughly 3-4 tokens per second. Text arriving at 30 TPS easily outpaces reading speed. At 15 TPS, medium-length responses start to feel sluggish. For chatbot applications, anything above 40 TPS provides a comfortable experience.

### Code Completion -- 50-100 TPS for Smooth Inline Suggestions

Inline code completion needs to finish generating a suggestion before the user moves on. The target window is roughly 200-400ms from cursor stop to suggestion display. At 80ms TTFT and a 20-token suggestion, you need about 100 TPS to finish within that window. At 50 TPS, the same suggestion takes 480ms total -- noticeably slow for an inline tool.

### Batch Document Processing -- Focus on Cost, Not TPS

For offline batch work (document summarization, data extraction, classification at scale), individual request TPS is less important than total throughput and cost per token. Higher TPS helps by reducing wall-clock time for the batch, but optimizing cost per token usually delivers more value than chasing higher per-request TPS.

### Voice AI -- 60+ TPS Required to Feed TTS in Real Time

Text-to-speech engines consume LLM output tokens and convert them to audio. TTS needs a token buffer ahead of current playback to avoid gaps. At typical speech rates of around 150 words per minute, you need the LLM to generate text approximately 1.5x faster than it's spoken -- roughly 60-80 TPS minimum.

Below that threshold, the TTS engine runs out of buffered text and the audio pauses. Voice AI is one application where high TPS is genuinely required, not just nice to have.

## How to Measure TPS in Your Own Environment

### Simple Python Benchmark Script

```python
import time
import openai

client = openai.OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

def benchmark_tps(model: str, prompt: str, max_tokens: int = 200) -> dict:
    start = time.perf_counter()
    first_token_time = None
    token_count = 0

    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1

    end = time.perf_counter()
    decode_duration = end - first_token_time if first_token_time else 0
    ttft = (first_token_time - start) if first_token_time else 0

    return {
        "model": model,
        "token_count": token_count,
        "ttft_ms": round(ttft * 1000, 1),
        "tps": round(token_count / decode_duration, 1) if decode_duration > 0 else 0,
        "total_duration_s": round(end - start, 2),
    }

result = benchmark_tps(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    prompt="Explain how transformer attention mechanisms work.",
    max_tokens=300,
)
print(result)
# Example output:
# {'model': 'meta-llama/...', 'token_count': 295, 'ttft_ms': 84.2, 'tps': 201.7, 'total_duration_s': 1.55}
```

Run this script several times and average the results -- individual measurements have meaningful variance.

### Stress Testing -- Measuring TPS Under Concurrent Load

To measure realistic P95 performance, send concurrent requests and collect statistics across all of them:

```python
import asyncio
import statistics
import openai

async_client = openai.AsyncOpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

async def benchmark_tps_async(model: str, prompt: str, max_tokens: int = 200) -> dict:
    start = time.perf_counter()
    first_token_time = None
    token_count = 0

    stream = await async_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )

    async for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1

    end = time.perf_counter()
    decode_duration = end - first_token_time if first_token_time else 0
    return {"tps": token_count / decode_duration if decode_duration > 0 else 0}

async def run_concurrent_benchmark(concurrency: int, model: str, prompt: str):
    tasks = [benchmark_tps_async(model, prompt) for _ in range(concurrency)]
    results = await asyncio.gather(*tasks)
    tps_values = sorted([r["tps"] for r in results if r["tps"] > 0])
    p95_index = max(0, int(len(tps_values) * 0.05))
    print(f"Concurrency {concurrency}: median={statistics.median(tps_values):.1f} TPS, P95={tps_values[p95_index]:.1f} TPS")

for concurrency in [1, 5, 10, 25, 50]:
    asyncio.run(run_concurrent_benchmark(
        concurrency=concurrency,
        model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
        prompt="Explain gradient descent in machine learning.",
    ))
```

Run this at increasing concurrency levels to understand how TPS degrades under load. The shape of that degradation curve tells you more than any single benchmark number.

### Interpreting P95 and P99 Latency in TPS Terms

When you collect TPS across many requests, sort the values and look at the bottom 5% (P95 worst case) and bottom 1% (P99). If your median is 150 TPS but P95 is 35 TPS, about 1 in 20 users is experiencing a noticeably degraded response. That usually indicates queuing delays at the server, not hardware limits. The fix is different depending on the cause: queuing problems call for spreading load across more instances, while hardware limits call for quantization or a different model.

## How to Maximize Tokens Per Second

### Use Quantized Models (INT4 / FP8)

If TPS is the bottleneck and quality tradeoffs are acceptable, moving from FP16 to FP8 precision is the first step. FP8 delivers 1.5-2x higher TPS with minimal quality change. INT4 pushes 3-4x TPS gains with a more significant quality tradeoff. For most production workloads, FP8 is the right default.

### Enable Continuous Batching

If you're self-hosting with vLLM or SGLang, continuous batching is enabled by default in recent versions -- but verify it's active. Static batching leaves GPU cycles idle while waiting for requests to finish before starting new ones. Continuous batching fills those idle cycles immediately, improving aggregate TPS substantially at the cost of slightly lower per-request TPS under concurrent load.

### Use a Dedicated Inference Provider vs DIY GPU

For most teams, a managed inference API delivers higher TPS than a self-managed GPU instance, because providers run infrastructure optimized specifically for serving at scale. Tuning vLLM, managing CUDA versions, handling memory fragmentation, and debugging throughput regressions is a full-time job.

If you're generating fewer than roughly 50 million tokens per day, a managed API is almost always more cost-effective and faster to iterate on than self-hosting. Above that volume, the break-even math starts shifting, but the operational overhead doesn't go away.

## FAQ

### How many tokens per second is ChatGPT?

OpenAI doesn't publish official TPS numbers for GPT-4o. Independent measurements consistently put GPT-4o in the 40-70 TPS range for typical requests, depending on server load at the time of measurement. Inference providers focused on open-source models and optimized hardware regularly deliver 150-200+ TPS for similarly-sized models.

### Does TPS vary between providers using the same model?

Yes, substantially. Two providers running Llama 4 8B can show 2-4x TPS differences based on hardware architecture, quantization precision, serving framework, and infrastructure-level optimizations. The model name alone doesn't tell you the performance you'll get. Always benchmark against your actual workload before committing to a provider.

### Is a higher TPS always better?

For a given use case, yes -- within reason. Chasing TPS at the expense of output quality (through aggressive INT4 quantization) or at the expense of TTFT (by over-prioritizing throughput over latency) can hurt user experience in ways that a raw TPS number won't reveal. Optimize TPS once you've confirmed your TTFT is acceptable for your use case, and only push quantization to the level your quality requirements allow.

---

If you want to run these benchmarks against your own workload, [General Compute's API](https://generalcompute.com) is OpenAI-compatible and offers some of the highest TPS numbers available for Llama 4 and other major open-source models. The benchmark script above works against our endpoint with a one-line base URL change.