KV Cache in LLM Inference: How It Works and Why It Matters

Every time a transformer generates a token, it needs to attend to every token that came before it. Without any optimization, this means recomputing the key and value vectors for the entire history on every single step. At sequence length 1,000, generating one token requires recomputing 1,000 sets of keys and values across every layer. At 10,000 tokens, the compute cost becomes prohibitive.

The KV cache solves this problem by storing those key and value vectors after they're first computed, so subsequent steps can reuse them. The name comes from the K (key) and V (value) matrices in the attention mechanism. The Q (query) matrix does not need to be cached because it's only ever computed for the current token.

Understanding the KV cache is useful because it sits at the center of most of the important tradeoffs in LLM serving: memory vs. compute, throughput vs. latency, batch size vs. context length.

Attention and Why K and V Are Special

In a standard multi-head attention layer, each token's hidden state produces three vectors: a query Q, a key K, and a value V. The attention score for the current token is computed by taking the dot product of its Q with the K of every other token in the sequence, passing through softmax, then using those scores to weight-sum the V vectors. The result is a new hidden state for the current token.

During autoregressive generation, you're computing one token at a time. Each new token needs to produce its Q and attend to all previous Ks, then use all previous Vs to form its output. Crucially, the Ks and Vs from previous tokens don't change between steps -- they depend only on the token and its position in the sequence. There's no reason to recompute them.

The KV cache stores each token's K and V vectors after they're computed, indexed by layer and position. When generating the next token, you compute Q, K, V for just the new token, append the new K and V to the cache, and run attention over the full cached K and V sequences. Compute per decode step stays constant regardless of sequence length.

Prefill and Decode: Two Different Problems

LLM inference has two distinct phases, and the KV cache behaves differently in each.

Prefill is the processing of the input prompt. The full prompt is available at once, so the model processes all tokens in parallel using regular matrix operations. At the end of prefill, the KV cache is populated with one entry per input token, per layer. Prefill is compute-bound: you're doing a lot of matrix multiplications in parallel and GPU utilization is high.

Decode is the generation phase, where tokens are produced one at a time. Each decode step extends the KV cache by one entry and runs attention over the entire cached sequence. Decode is memory-bandwidth-bound, not compute-bound. The arithmetic intensity is very low: read a large cache from HBM, do a small amount of computation, write one new entry. GPU utilization during decode is far lower than during prefill, especially for single-request workloads.

This asymmetry matters for serving. A user submitting a 32K-token document and requesting a 1,000-token summary will see a prefill phase that takes a fraction of the total time, followed by a decode phase that stretches out. Time-to-first-token (TTFT) is dominated by prefill. Token generation speed (tokens per second after the first token) is a decode concern.

How Much Memory Does the KV Cache Use?

The cache size for a single request depends on the model architecture and context length:

cache_bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes

The factor of 2 accounts for K and V. For Llama 3 70B with 80 layers, 8 KV heads (it uses GQA, covered below), head dimension 128, FP16, and a 32K context:

2 * 80 * 8 * 128 * 32,000 * 2 bytes = ~10.5 GB per request

A single H100 with 80GB of HBM can hold the model weights plus a small number of concurrent requests at that context length. Scale to 128K context and the per-request cache more than quadruples. This is the binding constraint in long-context serving, often more limiting than the weights themselves.

For a smaller model like Llama 3 8B (32 layers, 8 KV heads, 128 head dim), the cache at 32K context in FP16 is:

2 * 32 * 8 * 128 * 32,000 * 2 bytes = ~4.2 GB per request

Still substantial relative to the 8B model's roughly 16GB weight footprint at FP16. And this scales linearly with context: double the context length, double the cache.

Paged Attention: Managing the Cache Efficiently

A naive KV cache implementation pre-allocates memory per request based on the maximum possible context length. This wastes memory when requests end up shorter than expected, creates fragmentation when requests finish at different times, and prevents sharing memory across requests with common prefixes (like system prompts).

Paged attention, introduced in the vLLM paper, borrows the virtual memory paging concept from operating systems. Instead of a contiguous block per request, the cache is divided into fixed-size pages. Each page holds a fixed number of tokens' K and V vectors for a given layer. A page table maps logical positions (request ID, layer, token position) to physical pages in a shared pool.

The practical benefits are meaningful:

• No per-request pre-allocation. Pages are assigned as needed, one at a time.
• No wasted memory from early termination. Pages from a finished request return to the pool immediately.
• Prefix sharing. Multiple requests with the same system prompt can share the same physical pages for that prefix, with copy-on-write semantics if they diverge.

Modern serving frameworks -- vLLM, SGLang, TensorRT-LLM -- all implement paged attention or a close variant. It's effectively the standard for production serving. The attention kernel has to handle non-contiguous memory, which requires careful implementation, but the serving efficiency gains justify it.

MQA and GQA: Fewer KV Heads

The most direct way to shrink the KV cache is to have fewer keys and values to store. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) do this by reducing the number of independent KV heads while keeping query heads at full count.

In standard Multi-Head Attention (MHA), each attention head has its own Q, K, and V. With 64 heads, you store 64 K vectors and 64 V vectors per token per layer.

MQA (Shazeer, 2019) uses a single shared K and V pair across all query heads. You keep all 64 Q heads for expressivity, but cache only one K and one V. That's a 64x reduction in the KV cache, with a measurable quality regression on harder benchmarks.

GQA (Ainslie et al., 2023) is the practical middle ground: group query heads into g groups, with one shared K and V per group. Llama 3 uses 64 query heads and 8 KV heads (g = 8), giving an 8x cache reduction versus MHA. This preserves most of MHA's quality while making long-context serving economically viable. Essentially every modern open model uses GQA at this point.

The quality tradeoff comes from expressivity. In full MHA, each head independently specializes its keys and values. Sharing K and V across heads reduces that specialization. GQA lands in a range where the quality impact is small enough to be acceptable for almost all production workloads.

For a deeper look at how DeepSeek's Multi-Head Latent Attention (MLA) extends this idea using low-rank K/V projection for even larger cache reductions, see KV Cache Compression: MLA and Beyond.

KV Cache Quantization

Rather than changing the architecture, KV cache quantization keeps the same number of heads but stores them at lower precision.

FP16 is the standard baseline, but the cache is often a good candidate for quantization because the quality sensitivity to precision is lower than for the model weights themselves.

FP8 is the current production default where hardware supports it. Hopper-generation GPUs (H100, H200) have native FP8 support in their tensor cores, so you get 2x compression versus FP16 with essentially no dequantization overhead. The attention kernel can operate directly on FP8 values. Most modern serving frameworks support FP8 KV caches out of the box, and the quality impact is negligible for typical workloads.

INT8 gives the same 2x compression and works on older hardware, but the attention kernel has to dequantize on the fly. The overhead is small, and INT8 KV caches are well-supported across frameworks.

INT4 and lower are possible with careful per-group quantization schemes. KIVI (2024) demonstrated 2-bit quantization with per-group scales and acceptable quality loss. The memory savings are substantial (8x versus FP16), making ultra-low-bit caches worth watching for deployments with very long contexts or very high concurrency. Mainstream frameworks don't yet ship this by default.

KV cache quantization composes with GQA and paged attention. You can use GQA to have fewer heads, quantize those heads to FP8, and manage memory allocation with paged attention. The compression ratios stack multiplicatively.

What This Means in Practice

When you're designing a serving setup, the KV cache is often the binding constraint rather than the model weights. Here's why: model weights are static. You load them once and they stay in memory. The KV cache is dynamic -- it grows with context length, varies by request, and competes with other concurrent requests for the same HBM pool.

The number of concurrent requests you can serve depends on:

• The maximum context length you support
• The number of KV heads in the model (model architecture)
• The dtype you're using for the cache (quantization)
• How efficiently you're managing memory (paged vs. contiguous allocation)

If your model supports 128K context but your average request uses 4K, contiguous pre-allocation wastes most of your memory budget. Paged attention fixes this. If you're on an H100 and using FP16 for the KV cache, switching to FP8 doubles your effective cache capacity at no quality cost. If you're selecting a base model and haven't checked whether it uses GQA, the cache size could be 8x larger than necessary.

Long-context deployments amplify all of this. A model serving 1M-token contexts needs very careful cache management -- the per-request footprint can exceed the weight footprint by a large margin. This is the engineering problem that motivated both MLA in DeepSeek-V3 and the focus on FP8 KV caches across the industry.

Here's a rough comparison to make the stakes concrete. Suppose you're serving a 70B model on a node with 640GB of HBM (8xH100) and want to support 32K context:

| Configuration | Cache per request | Concurrent requests at 50% HBM budget | |---|---|---| | MHA, FP16 | ~84 GB | ~3 | | GQA (8 KV heads), FP16 | ~10.5 GB | ~30 | | GQA, FP8 | ~5.2 GB | ~61 |

The model weights take roughly 140GB at BF16 (or less with quantization), leaving around 320GB for cache if you budget 50% for it. The table above shows how architecture and quantization choices translate directly into serving capacity.

Putting It Together

The KV cache is what makes autoregressive generation tractable at practical sequence lengths. Without it, each decode step would require recomputing all prior context from scratch. With it, each step reads the existing cache from memory, appends one entry, and moves on. Compute per step is constant; memory grows linearly.

The techniques built on top -- paged attention, GQA, quantization -- are all about using that memory more efficiently. They allow more concurrent requests, longer contexts, or both, within the same hardware budget. They don't change the fundamental mechanics, just the economics.

If you're debugging why inference costs more than expected at longer context lengths, or why throughput drops off when concurrency increases, measuring KV cache utilization is usually the right place to start. The numbers often explain the bottleneck more directly than any other metric.

General Compute's infrastructure is designed for high-throughput, low-latency inference across long contexts. If you're working with memory-intensive serving workloads and want to evaluate the tradeoffs, you can get started with the API at generalcompute.com.