Scaling Laws in LLMs: What They Mean for Inference Cost in Production

Markdown body

Scaling laws are the closest thing machine learning has to first principles. They describe a predictable relationship between model size, training data, compute budget, and the quality of the resulting model. If you're choosing which model to run in production or trying to estimate what a smaller model will cost you in accuracy, understanding scaling laws will save you a lot of expensive trial and error.

This post covers what the major scaling law papers actually say, how to interpret them for inference decisions (not just training), and what the shift toward smaller, more compute-optimal models means for your cost-per-query at scale.

## What Scaling Laws Say (and What They Don't)

The foundational work here is the 2020 paper from Kaplan et al. at OpenAI, often called the "GPT-3 scaling laws paper." The core finding: language model loss improves as a power law with respect to three variables -- model size (number of parameters), training compute (total FLOPs), and dataset size (number of tokens). Double any of these while holding the others fixed and you get a predictable, measurable improvement in perplexity.

Crucially, the Kaplan paper found that model size mattered more than dataset size when you had a fixed compute budget. Their recommendation: train very large models on relatively fewer tokens. This reasoning drove the design of GPT-3 (175B parameters, trained on roughly 300B tokens).

Then in 2022, DeepMind published "Training Compute-Optimal Large Language Models" -- commonly called the Chinchilla paper. Hoffmann et al. trained over 400 models across a wide range of sizes and token counts and reached a different conclusion: **Kaplan's models were significantly undertrained**. The Chinchilla result: for a fixed compute budget, you should train a model with roughly 20 tokens per parameter. A 10B parameter model should see 200B tokens. A 70B model should see about 1.4 trillion tokens.

GPT-3 at 175B was trained on 300B tokens. By Chinchilla's math, a 70B model trained on the same compute budget would outperform it, because you're spending those FLOPs more efficiently.

Chinchilla (the actual model) proved this out, matching or beating models two to four times its size on benchmarks. Llama 2 and Llama 3, as well as Mistral's models, followed the same principle -- train smaller models longer on more tokens.

## The Training vs. Inference Trade-off

Here's where scaling laws connect directly to your production infrastructure costs.

Kaplan's original framing was about training efficiency: given a compute budget, how do you minimize loss? But inference has a very different cost structure. When you deploy a model, you pay per forward pass. The cost of a single inference call scales roughly linearly with the number of parameters -- a 70B model costs roughly twice as much per token to serve as a 35B model at similar batch sizes.

So the Chinchilla result has a practical implication that goes beyond benchmarks: smaller compute-optimal models are cheaper to run and, for the same training compute spent, are equally capable. If Chinchilla-style training means a 7B model can do what a 30B undertrained model could do, your inference costs drop by 4x.

This is why the open-source ecosystem largely moved toward Chinchilla-optimal training after 2022. Llama 2 7B was trained on 2 trillion tokens. Llama 3 8B was trained on 15 trillion. These models punch above their weight in benchmarks precisely because they follow better compute allocation.

## How to Use Scaling Laws for Model Selection

The practical question for most teams is: given the task, what's the smallest model I can get away with?

Scaling laws give you a useful mental model here, but they're not a substitute for benchmarking. What they tell you:

**Capability scales with effective compute, not raw parameters.** A 7B model trained on 15T tokens likely outperforms a 13B model trained on 1T tokens. When comparing models from different training runs, parameter count alone is a poor proxy for quality.

**Task difficulty matters.** Scaling laws predict average loss on a language modeling objective. Your specific task -- code generation, structured output, multi-step reasoning -- may not scale the same way. Coding and math tasks tend to require more parameters to solve reliably, even holding training compute constant.

**Beyond a certain scale, gains flatten.** The power law curves are not linear -- they have diminishing returns. Going from 7B to 70B often gives you meaningful gains on hard reasoning tasks. Going from 70B to 700B gives you much less, for ten times the inference cost. Unless you specifically need frontier-model capabilities, you're often paying for marginal improvements.

A reasonable decision framework:

- For classification, extraction, summarization, or simple Q&A: a well-trained 7B or 8B model is usually sufficient and costs a fraction of a 70B model to serve.
- For complex reasoning, multi-step planning, or tasks that require broad knowledge: 30B to 70B range is the practical sweet spot.
- For frontier tasks that require SOTA performance: you're looking at 70B+ dense models or large MoE models like DeepSeek V3, which achieve high capability through sparse routing rather than dense scale.

## Small Language Models and the Production Case

One consequence of Chinchilla-optimal training is that "small" doesn't mean "bad" anymore. Models like Phi-3 Mini (3.8B), Qwen 2.5 3B, and Llama 3.2 3B are trained on enormous token counts relative to their size. On many benchmarks they match models that were considered large just two years ago.

For production inference, this matters a lot. Small models have:

**Lower latency.** Fewer parameters mean fewer matrix multiplications per token. At the same hardware, a 3B model generates tokens roughly five to ten times faster than a 70B model. If you're building voice AI or anything with real-time requirements, this isn't a nice-to-have -- it's required.

**Higher throughput.** You can fit more concurrent requests on the same GPU. A 3B model on a single A100 can handle 50+ concurrent sequences; a 70B model needs multiple GPUs just to load the weights.

**Lower cost per query.** Combining lower hardware requirements with higher throughput, the effective cost per 1M tokens for a 3B model is roughly 10 to 20 times cheaper than a 70B model on equivalent hardware.

The practical catch is that small models fail more on hard tasks. If your application involves complex multi-step reasoning, legal analysis, nuanced code generation, or long-horizon planning, you'll likely see meaningful quality degradation at 3-8B. Benchmark your actual task, not just generic leaderboard scores.

## Inference Cost Estimation Using Scaling Law Intuitions

You can use scaling law thinking to estimate what you'll pay before you run a single query.

The cost of a forward pass through a transformer scales roughly as:

```
cost ≈ 2 * num_parameters * num_tokens
```

in FLOPs (the factor of 2 accounts for multiply-accumulate operations). At a fixed hardware throughput (tokens/second/GPU), you can convert this to latency and cost.

For rough budget planning:

| Model Size | Relative FLOP Cost per Token | Typical TPS (single A100) |
|------------|------------------------------|---------------------------|
| 3-8B       | 1x                           | 80-150 tok/s              |
| 13-14B     | 2-3x                         | 40-70 tok/s               |
| 30-34B     | 5-6x                         | 15-30 tok/s               |
| 70B        | 10x                          | 8-15 tok/s                |
| 405B+      | 50x+                         | 1-3 tok/s (multi-GPU)     |

These are rough figures that vary significantly with quantization, batch size, and hardware. But the relative ratios hold up well enough for planning.

If you're generating 100M tokens per day:

- At 7B (with good inference infra): you might pay $0.10-0.20 per 1M tokens
- At 70B: that same output costs 5-10x more in compute

At scale, the model size choice is one of the highest-leverage cost decisions you'll make.

## Quantization and the Effective Scaling Budget

Scaling laws assume you're running models in full precision (typically BF16 or FP16). Quantization complicates this but in a mostly favorable direction.

INT4 or INT8 quantization reduces memory bandwidth requirements, which is often the actual bottleneck in inference (not raw compute). This means you can run a larger model at a given throughput target, effectively shifting the cost curve.

A 70B model quantized to INT4 fits in roughly 40GB of VRAM -- a single high-end GPU. Running it at INT4 costs about the same per token as running a 30B model at FP16. So quantization partially offsets the cost disadvantage of larger models, as long as your quality tolerance includes the small perplexity hit from reduced precision.

The practical takeaway: when comparing models for production deployment, compare:

1. The benchmark results at your target quantization level (not just FP16)
2. The throughput numbers at that quantization level on your target hardware
3. The quality on your actual task, not just generic benchmarks

Scaling laws predict quality at training precision. Quantization introduces a small, predictable degradation that's usually worth the infrastructure savings.

## What "Compute-Optimal" Means at Inference Time

There's an emerging extension of Chinchilla reasoning to inference: the idea of "inference-optimal" models, sometimes called test-time compute scaling.

The basic insight: if you know how many inference calls you'll make, you can treat inference FLOPs as part of your total compute budget and optimize accordingly. A smaller model that you call five times might be cheaper than a large model called once, if the aggregated outputs reach the same quality.

This is one motivation behind chain-of-thought prompting and reasoning models. Llama 3 8B with a careful reasoning prompt can sometimes match a 70B model on specific tasks, at a fraction of the inference cost -- though this doesn't generalize to all tasks and adds latency from longer output sequences.

The more principled version of this is speculative decoding, where a small draft model generates candidate tokens and a larger verifier model checks them in parallel. The draft model does most of the work; the verifier runs less often. You get output that matches the larger model's quality at throughput closer to the small model's speed. This is a direct application of inference-budget thinking derived from scaling law intuitions.

## Practical Recommendations

If you're making model selection decisions for a production system, here's how to apply this:

**Don't treat parameter count as a quality signal in isolation.** A Llama 3 8B outperforms many older 30B models. Check training token counts and benchmark results for your specific task category.

**Start small and move up only when you see quality failures.** Deploy a 7-8B model first. Measure task performance on a representative sample. Escalate to 30-70B only for the failure cases, using a router if needed.

**Factor in quantization from the start.** INT8 is nearly lossless for most tasks. INT4 is acceptable for many workloads. Plan your memory budget around quantized model sizes so you're not over-provisioning hardware for FP16 weights you don't need.

**Benchmark your actual task.** Scaling laws predict average capability on language modeling. Your task -- structured JSON extraction, code completion with a specific style guide, domain-specific summarization -- may behave very differently from MMLU or HumanEval. Run evals on real examples before committing to a model.

**Consider total cost of ownership.** A smaller model at higher request rates will sometimes require more instances than a larger model with caching and batching optimizations. Model the full system, not just the per-token cost.

---

If you want to run these experiments without managing your own GPU cluster, [General Compute's API](https://generalcompute.com) gives you access to a range of open-weight models -- from sub-10B to 70B+ -- on fast inference infrastructure with OpenAI-compatible endpoints. You can benchmark models side by side, measure latency at your target throughput, and find the right size for your workload before committing to a deployment decision.