Instruction-Tuned vs Base Models: Does Fine-Tuning Cost You Speed?
A question comes up regularly when teams are picking a model for production: should we use the base model or the instruct checkpoint? The instruct version is the obvious choice for most tasks, but there's a persistent intuition that fine-tuning "adds something" to the model and therefore makes it slower. This post works through whether that's true, where real speed differences actually come from, and how to benchmark two checkpoints of the same architecture fairly.
What Fine-Tuning Actually Changes
Instruction tuning (and its variants like RLHF, DPO, and ORPO) is a fine-tuning process that adjusts the model's weights to align it with human preferences and instruction-following behavior. What it does not do is change the model's architecture.
A fine-tuned checkpoint has:
- The same number of layers
- The same number of attention heads and KV heads
- The same hidden dimension
- The same vocabulary size (in most cases)
- The same parameter count
The model is structurally identical to its base version. Inference speed is determined primarily by the number of parameters and how you're serving them -- the same matrix multiplications happen in the same order for every forward pass. Changing the values in those weight matrices doesn't change the FLOPs required per token.
So the short answer is: all else being equal, an instruction-tuned model and its base counterpart run at identical speed.
The longer answer is that "all else being equal" is doing real work there.
Where Real Speed Differences Appear
Chat Templates Add Input Tokens
The most common source of measured speed differences between base and instruct checkpoints isn't architectural -- it's context length.
Instruction-tuned models expect to be called with a chat template. This template wraps your prompt in role-tagged messages and usually includes a system prompt. On top of that, many teams add a system prompt explaining the model's purpose, constraints, or persona.
When you prompt a base model with "Summarize this article:" and then a 500-token article, your total input is about 510 tokens. When you prompt an instruct model with the same task, you might have:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Summarize this article:
[500 tokens of article content]<|eot_id|><|start_header_id|>assistant<|end_header_id|>
That system prompt and chat template scaffolding adds 30-60 tokens depending on the model family. For a single request, the extra prefill cost is small. For high-throughput batch jobs, those tokens add up.
The performance difference is not inherent to the checkpoint -- it's a consequence of how you're using it.
LoRA Adapters vs Merged Checkpoints
There's a meaningful distinction between fine-tuned models distributed as merged checkpoints and those distributed as LoRA adapters meant to be loaded on top of a base model.
A merged checkpoint has the LoRA weights incorporated into the base weights. It runs at the same speed as any other checkpoint of that architecture. There's no overhead at inference time.
Serving a LoRA adapter separately is different. Frameworks like vLLM support "PEFT"-style adapter serving where the base model is loaded once and adapters are applied per-request. This is efficient for serving many fine-tuned variants of the same base model, but it does add a small per-request overhead: the adapter weights need to be applied on top of the base weights during the forward pass.
For most practical purposes, merged checkpoints are the norm when a model is released publicly. If you're downloading Llama 3.1 8B Instruct, you're getting a merged checkpoint, not a detached adapter.
RoPE Scaling and Extended Context Windows
Some fine-tuned models include modifications to rotary position embeddings (RoPE) to extend the effective context window beyond what the base model was trained on. Llama 3.1 extended the context window to 128K tokens using RoPE scaling, compared to Llama 3's 8K window.
Extended context affects inference in two ways. First, the KV cache grows with context length -- a 128K context window requires substantially more GPU memory for KV cache than an 8K window. Second, the position embedding computation itself is slightly different, though this is a negligible cost.
If you're comparing a base model at its native context window to a fine-tuned model at an extended context window, you may see differences in memory pressure and effective batch size, which indirectly affects throughput. This is a characteristic of the training recipe, not the instruction tuning itself.
Vocabulary Changes
Most instruction-tuned models don't change the vocabulary. But some fine-tunes, particularly multilingual adaptations or models trained with extended tokenizers, add tokens to the vocabulary. A larger vocabulary means a larger embedding table and a larger final projection layer. For very large vocabulary additions, this can increase the model's parameter count slightly and change memory footprint.
In practice, the major instruction-tuned checkpoints (Llama, Qwen, Mistral) keep the same vocabulary as their base models, so this isn't a factor in typical comparisons.
Benchmark: Llama 3.1 8B vs Llama 3.1 8B Instruct
To make this concrete, here's what you should expect when running Llama 3.1 8B and Llama 3.1 8B Instruct at the same input/output configuration on GPU:
| Configuration | Tokens/s (approx) | TTFT (approx) | |---|---|---| | Base model, 512 input / 256 output | ~2,800 | ~18ms | | Instruct model, 512 input / 256 output | ~2,800 | ~18ms | | Instruct model, 512+50 input (chat template) / 256 output | ~2,750 | ~20ms | | Instruct model, 512+50+150 input (with system prompt) / 256 output | ~2,650 | ~24ms |
These numbers are approximate and will vary by hardware, batch size, and serving framework. The key pattern is that the base vs. instruct checkpoint itself shows no throughput difference when input lengths are matched. The small differences in the table come entirely from the extra tokens added by the chat template and system prompt.
TTFT increases roughly linearly with input token count -- the prefill phase processes all input tokens in parallel, so more tokens means more compute before the first output token is generated. At 128K context, the prefill cost becomes significant regardless of whether you're using a base or instruct model.
Quantization Behaves Differently Across Checkpoints
Even though base and instruct checkpoints have the same architecture, quantization can affect them differently. Here's why.
GPTQ and AWQ calibrate quantization parameters based on a sample dataset -- they pick the quantization scale that minimizes output error on that data. Instruction-tuned models have different weight distributions than base models because the fine-tuning process shifts weights toward generating helpful, aligned responses. The calibration dataset matters more for instruction-tuned models if the calibration examples don't represent the model's expected usage pattern.
In practice, quantized instruct models are usually calibrated with instruction-format examples (chat templates, user/assistant turns) to get the best quality out of 4-bit or 8-bit quantization. Using a base model calibration for an instruct checkpoint, or vice versa, can cause perplexity degradation.
This doesn't affect throughput -- a 4-bit GPTQ model runs at the same tokens/second regardless of whether it's a base or instruct checkpoint. But it affects quality, which can matter when you're trying to understand "speed vs. quality" tradeoffs across checkpoints.
FP8, which is used by higher-end serving infrastructure (H100, H200), is more robust to calibration choice because it has higher precision than INT4 and the quantization error is correspondingly smaller. The base vs. instruct distinction matters less for FP8 quality than for 4-bit formats.
The Actual Decision: When to Use Each
Use the instruction-tuned checkpoint for any task that involves responding to user requests, following instructions, or generating dialogue. The fine-tuning makes the model substantially more useful without costing you throughput. Most production applications -- chatbots, assistants, coding helpers, document processors -- should default to the instruct variant.
Use the base checkpoint when:
- You're continuing pre-training or doing your own instruction tuning from scratch
- Your task is few-shot in-context learning where you're providing many examples in the prompt (base models often follow few-shot patterns more literally)
- You're evaluating raw model capability before fine-tuning for a specialized domain
- You're building an evaluation baseline and want to isolate the contribution of instruction tuning
For inference speed specifically, the choice between base and instruct should not be driven by throughput concerns. If you're measuring a real-world speed difference, look at whether you're comparing at equal input lengths first.
Benchmarking Two Checkpoints Fairly
When you run your own benchmarks comparing model variants, a few practices prevent apples-to-oranges comparisons.
Fix input and output lengths. Use a fixed number of input tokens (padding if necessary) and a fixed number of generation tokens. Varying input lengths across runs conflates the prefill latency with what you're trying to measure.
Account for chat templates in instruct models. When benchmarking instruct models, include the chat template tokens in your input length count. If your real workload includes a 150-token system prompt, include it in the benchmark.
Use the same precision. Base and instruct models at FP16 compare fairly. FP16 vs INT4 do not -- quantization format is a separate variable.
Run at realistic batch sizes. Throughput benchmarks on a single request look different from batch sizes of 16 or 64. If your production workload serves concurrent requests, benchmark at representative batch sizes.
Repeat runs and report distributions. GPU performance has variance from memory allocation, thermal state, and scheduling. A single measurement is noisy. Report p50 and p99, not just the best run.
A minimal Python benchmark you can adapt:
import time import statistics from openai import OpenAI client = OpenAI( base_url="https://api.generalcompute.com/v1", api_key="YOUR_API_KEY" ) def benchmark_model(model_id, prompt, n_runs=20, max_tokens=200): latencies = [] for _ in range(n_runs): start = time.perf_counter() response = client.chat.completions.create( model=model_id, messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, ) elapsed = time.perf_counter() - start tokens_out = response.usage.completion_tokens latencies.append(elapsed / tokens_out * 1000) # ms per token return { "p50_ms_per_token": statistics.median(latencies), "p99_ms_per_token": sorted(latencies)[int(0.99 * len(latencies))], } # Use a fixed prompt length for both models fixed_prompt = "Explain the following concept in detail: " + "word " * 100 results_base = benchmark_model("meta-llama/Llama-3.1-8B", fixed_prompt) results_instruct = benchmark_model("meta-llama/Llama-3.1-8B-Instruct", fixed_prompt) print("Base:", results_base) print("Instruct:", results_instruct)
Note that this benchmark sends the same raw text to both models. For a fair real-world comparison of what you'd actually deploy, you'd wrap the instruct model call with the chat template, which would increase its input length relative to the base call.
Summary
The architecture is the same, so the inference cost is the same. Fine-tuning changes weights, not structure. The sources of real-world speed differences between base and instruct checkpoints are:
- Extra tokens from chat templates and system prompts (increases TTFT and slightly reduces throughput)
- LoRA adapter overhead when not merged (small per-request cost)
- Extended context window support (affects KV cache memory pressure)
- Vocabulary additions (rare, but changes parameter count at the embedding layer)
For production serving, run the instruct model. It's more capable for real tasks, runs at the same throughput when input lengths are matched, and the fine-tuning overhead is negligible compared to the capability gain.
If you're evaluating models and want to isolate fine-tuning's impact on speed vs. quality, General Compute's API supports both base and instruct variants of the major open-source families, making it straightforward to run the comparison at scale without managing GPU infrastructure.