We secured a $400M debt facility with Upper90 to scale inference compute.Read
benchmarksthroughputinferencecostmodelsbatch-inference

Batch Inference Throughput: Which Models Maximize Tokens-per-Dollar?

General Compute·

Latency benchmarks dominate most model comparisons. Time to first token, decode speed at single-digit batch sizes, interactive response feel -- these metrics matter a lot for user-facing applications. But a substantial share of LLM workloads are not user-facing at all.

Document classification, data extraction, content moderation, synthetic data generation, embedding pipelines, nightly summarization jobs -- these workloads are batch-oriented. They run against a queue, they tolerate a few seconds of latency per request, and what actually drives cost is throughput: how many tokens does the model produce per dollar spent?

This post benchmarks the models most commonly used for batch workloads and answers that question directly.

What Throughput-Focused Workloads Actually Need

When you're running a batch job that processes 50 million tokens overnight, the metrics that matter shift:

  • Tokens per second at real batch sizes: Not batch-size-1 latency, but throughput under the concurrent request volumes you'll actually send.
  • Effective cost per million output tokens: Determined by throughput and infrastructure cost combined.
  • Quality floor: You need the model to be good enough for the task, but you generally don't need the frontier model -- you need the cheapest model that crosses the quality threshold for your specific use case.

The practical question is: which model produces the most usable output per dollar?

Methodology

All tests were run on GeneralCompute infrastructure. We measured each model at batch sizes of 1, 8, 32, and 64 concurrent requests. Input context was fixed at 2,048 tokens per request, with 512 output tokens requested per request. Three runs per configuration, median reported.

Models tested:

  • Qwen2.5 7B Instruct -- 7B dense transformer, aggressive GQA
  • Phi-4 14B -- 14B dense, optimized for instruction following
  • Qwen2.5 72B Instruct -- 72B dense, 8 KV heads (GQA)
  • Llama 3.1 70B Instruct -- 70B dense, GQA
  • Mixtral 8x7B -- MoE, approximately 13B active parameters per token
  • Llama 4 Scout -- MoE, approximately 17B active parameters per token
  • DeepSeek V3 -- MoE, approximately 37B active parameters per token

Quality was not directly measured in this benchmark. That's deliberate -- quality is task-dependent, and the best throughput number in the world does not matter if the model cannot handle your specific task. Use the quality benchmarks from our other posts (MMLU, HumanEval, domain-specific evals) to determine which models are viable, then use this data to pick the most cost-efficient one among them.

Throughput Results by Batch Size

Batch Size 1 (Baseline)

Single-request throughput matters for latency-sensitive workflows and as a baseline for understanding how batching helps.

| Model | Parameters | Decode Speed (tok/s) | |---|---|---| | Qwen2.5 7B | 7B | 97 tok/s | | Phi-4 14B | 14B | 61 tok/s | | Mixtral 8x7B | ~13B active | 55 tok/s | | Llama 4 Scout | ~17B active | 48 tok/s | | Llama 3.1 70B | 70B | 22 tok/s | | Qwen2.5 72B | 72B | 26 tok/s | | DeepSeek V3 | ~37B active | 31 tok/s |

At batch size 1, MoE models are fast but not dramatically so compared to dense models at similar quality levels. The more interesting story emerges as batch size increases.

Batch Size 32 (Production Batch Workload)

Batch size 32 represents a realistic concurrent request volume for production batch pipelines. At this level, continuous batching fills the GPU compute units efficiently and throughput gains from batching start to compound.

| Model | Total Throughput (tok/s) | Per-Request Speed (tok/s) | |---|---|---| | Qwen2.5 7B | 2,810 tok/s | 88 tok/s | | Phi-4 14B | 1,680 tok/s | 52 tok/s | | Mixtral 8x7B | 1,510 tok/s | 47 tok/s | | Llama 4 Scout | 1,380 tok/s | 43 tok/s | | DeepSeek V3 | 1,140 tok/s | 36 tok/s | | Llama 3.1 70B | 580 tok/s | 18 tok/s | | Qwen2.5 72B | 690 tok/s | 22 tok/s |

Total throughput here is what matters for a batch pipeline: it determines how long your job takes. Qwen2.5 7B at batch size 32 produces 2,810 tokens per second. Llama 3.1 70B produces 580 -- about 4.8x less.

Batch Size 64 (High-Concurrency Batch)

At batch size 64, we start seeing memory pressure effects on some models. Larger models require more KV cache per request, and at 64 concurrent requests the accumulated KV cache can push into memory limits, causing scheduler queuing that reduces effective throughput.

| Model | Total Throughput (tok/s) | Throughput Gain vs B32 | |---|---|---| | Qwen2.5 7B | 3,280 tok/s | +17% | | Phi-4 14B | 1,940 tok/s | +15% | | Mixtral 8x7B | 1,760 tok/s | +16% | | Llama 4 Scout | 1,590 tok/s | +15% | | DeepSeek V3 | 1,290 tok/s | +13% | | Llama 3.1 70B | 620 tok/s | +7% | | Qwen2.5 72B | 730 tok/s | +6% |

Smaller models continue to scale nearly linearly from B32 to B64. Larger dense models show diminishing returns, gaining only 6-7% because KV cache memory is becoming the bottleneck. For these models, B32 is often near the practical limit of efficient batch size -- pushing higher adds latency without proportional throughput gains.

Tokens-per-Dollar Comparison

Throughput numbers alone do not tell the full story. What you actually care about is cost efficiency: how many tokens do you get per dollar? We calculated effective tokens-per-dollar by combining throughput with compute cost at each batch size.

Prices are in output tokens per dollar, at batch size 32:

| Model | $/M Output Tokens | M Tokens/Dollar | |---|---|---| | Qwen2.5 7B | $0.18 | 5.56M | | Phi-4 14B | $0.31 | 3.23M | | Mixtral 8x7B | $0.28 | 3.57M | | Llama 4 Scout | $0.35 | 2.86M | | DeepSeek V3 | $0.42 | 2.38M | | Qwen2.5 72B | $0.82 | 1.22M | | Llama 3.1 70B | $0.79 | 1.27M |

Qwen2.5 7B produces 5.56 million output tokens per dollar. Llama 3.1 70B produces 1.27 million per dollar -- about 4.4x less cost-efficient for the same output volume. If 7B quality clears your bar, running it instead of a 70B model cuts your batch processing cost by roughly 80%.

MoE models land in an interesting middle position. Mixtral 8x7B at 3.57M tokens/dollar and Llama 4 Scout at 2.86M tokens/dollar offer better cost efficiency than dense 70B models at noticeably better quality than 7B. For tasks where 7B falls short but full 70B capability is not required, these are often the practical sweet spot.

What Drives the Differences

Model size is the primary cost driver. Larger models require more memory bandwidth per token generated, and memory bandwidth is the fundamental bottleneck during decode. This translates directly to lower throughput and higher cost per token.

GQA compression helps larger models more than smaller ones. Qwen2.5 72B uses only 8 KV heads versus Llama 3.1 70B's 8 GQA heads, but Qwen's attention architecture achieves higher throughput at large batch sizes because the KV cache per request is smaller. At batch size 32, Qwen2.5 72B is about 19% faster than Llama 3.1 70B despite similar parameter counts. Aggressive GQA is a meaningful throughput lever.

MoE models decouple parameter count from compute. Mixtral 8x7B has 46.7B total parameters but only activates about 13B per token. The router selects two of eight expert layers for each token, so per-token compute is similar to a 13B dense model, while the model can express significantly more capacity through its specialized experts. At batch sizes common in production workloads, this translates to throughput comparable to 13B models at quality closer to 34B-class dense models.

Continuous batching is essential to realize batch gains. The throughput numbers above assume continuous batching: new requests enter the batch as completed requests leave, keeping GPU utilization high. Without it, you would see batch-size-1 throughput regardless of how many requests you send concurrently. Modern inference servers including vLLM and SGLang implement this by default, as does GeneralCompute's hosted API.

KV cache memory caps effective batch size. At batch size 64, larger models stop scaling because memory is full of KV cache entries. The practical maximum batch size for a 70B model on typical GPU configurations is lower than for a 7B model, which means the throughput gap between small and large models widens at high concurrency rather than narrowing.

Choosing a Model for Your Batch Workload

The decision comes down to where your task sits on the quality-vs-cost spectrum.

If Qwen2.5 7B quality is sufficient, it is the clear choice for throughput-focused workloads. 5.56M tokens per dollar is difficult to beat. Tasks in this category include text classification, basic extraction from structured documents, short-form summarization where precision matters less than volume, and simple code generation.

If 7B is borderline, test Phi-4 14B and Mixtral 8x7B before jumping to 70B. Phi-4 14B punches above its parameter count on instruction following and reasoning tasks. Mixtral 8x7B offers different strengths, particularly on multilingual text and tasks where diverse expertise benefits. Both are around 3-3.5x more cost-efficient than a 70B dense model.

If you need 70B-class quality, prefer Qwen2.5 72B over Llama 3.1 70B for throughput-sensitive workloads. The GQA architecture gives Qwen roughly 20% higher throughput at large batch sizes, and at the same quality tier that translates directly to lower cost per million tokens.

For tasks requiring the highest reasoning capability, DeepSeek V3's MoE architecture gives you frontier-adjacent quality at 2.38M tokens/dollar -- substantially more cost-efficient than comparable dense frontier models, though more expensive than the models above it in this list.

Structuring Batch Jobs for Maximum Efficiency

A few practical points on getting the most out of batch inference:

Set realistic batch sizes. Sending all requests at once with a large batch is more efficient than sending them one at a time. If your pipeline allows it, queue work and send in batches of 32-64 requests. For continuous workloads, this happens naturally; for one-off batch jobs, it may require buffering.

Tune output length. Tokens generated directly determine cost. If your task requires a classification (one of N labels) or a short answer, constrain max_tokens accordingly. Generating 512 tokens when you need 50 wastes 90% of your token budget.

Use prompt caching for shared prefixes. If all requests in your batch share a long system prompt, prefix caching amortizes the cost of processing that prefix across the entire batch. For a 1,000-token system prompt repeated across 100,000 requests, prefix caching avoids processing 100 million tokens of duplicated input.

Profile before committing to a model. Run 100-1,000 representative examples from your actual task through each candidate model. Quality benchmarks on academic evals are a starting point, but the quality floor that matters is the one specific to your data and acceptance criteria.

Running Batch Workloads on GeneralCompute

GeneralCompute's API supports batch workloads through the standard OpenAI-compatible completions endpoint. Continuous batching is handled server-side, so you can send concurrent requests without managing batching logic yourself.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

async def process_item(text: str) -> str:
    response = await client.chat.completions.create(
        model="qwen2.5-7b-instruct",
        messages=[{"role": "user", "content": text}],
        max_tokens=256
    )
    return response.choices[0].message.content

async def process_batch(items: list[str], concurrency: int = 32) -> list[str]:
    semaphore = asyncio.Semaphore(concurrency)
    
    async def bounded(item):
        async with semaphore:
            return await process_item(item)
    
    return await asyncio.gather(*[bounded(item) for item in items])

# Example: process 10,000 documents
with open("documents.txt") as f:
    documents = f.readlines()

results = asyncio.run(process_batch(documents, concurrency=32))

This pattern sends up to 32 concurrent requests, which matches the batch size where Qwen2.5 7B achieves 2,810 tokens per second in the benchmarks above. Adjusting the concurrency parameter up or down lets you tune throughput vs per-request latency for your workload.

For larger batch jobs, the GeneralCompute API does not enforce per-minute throttling on output tokens the way some providers do, which means a sustained batch job at high concurrency will maintain throughput rather than hitting rate limit walls mid-job.

See the GeneralCompute docs for current model pricing and available models. Prices change as hardware efficiency improves, so check the latest numbers before budgeting large batch jobs.

ModeHumanAgent