How Transformer Architecture Determines Inference Speed and Memory Usage

When you pick a model for production, you're not just choosing a quality level -- you're choosing a set of hardware constraints. The architecture decisions made when a model was designed and trained determine how much memory it occupies, how many FLOPs each token costs, and where your serving bottleneck will be. Those decisions are frozen into the weights. You can change some things at inference time (quantization, batching, hardware selection), but the fundamental structure is fixed.

This post works through the pieces of a transformer that matter most for inference: what each component does, how much it costs, and what the formulas look like with real numbers.

Why Understanding Transformer Architecture Matters for Inference

How Architecture Choices Made at Training Time Constrain Inference

Every hyperparameter in a model's design has a corresponding inference cost. Hidden dimension determines how much memory bandwidth each layer needs. Number of layers determines how many sequential kernel launches happen per token. KV head count determines how much memory the cache consumes per request. Sequence length at training time (not at inference time) determines nothing -- but the context window the model was trained to handle sets the quadratic attention cost you'll face at long inputs.

None of this is adjustable without retraining. You can quantize weights to reduce memory footprint, but you can't change how many layers a model has or how its attention heads are organized. Architecture selection is a permanent inference decision.

The Three Inference Bottlenecks

Most inference workloads are constrained by one of three things:

Compute (FLOPs) is the raw number of multiply-accumulate operations needed per token. This scales with model size, primarily driven by the feed-forward network (FFN) layers. Dense models are more compute-bound than sparse MoE models at equivalent quality.

Memory bandwidth is how fast the GPU can read weights and KV cache from HBM into compute units. The decode phase of generation is almost always memory-bandwidth-bound -- you do very little compute per byte of data read, so GPU utilization is low regardless of how fast the compute cores are.

Latency is a function of both of the above, plus kernel launch overhead, network round-trips in distributed setups, and scheduler latency in the serving framework. Reducing latency often means trading throughput.

Understanding which bottleneck applies to your workload tells you which optimization levers are actually worth pulling.

The Transformer Block: What Actually Runs During Inference

A standard transformer layer has a few distinct components that run sequentially per token.

Layer Normalization

Layer norm normalizes the hidden state vector before attention and FFN. It's computationally cheap -- linear in hidden dimension -- but it adds a read-modify-write cycle over the full hidden state at every layer. In a 128-layer model running at high concurrency, this adds up.

Multi-Head Attention: The Memory Bandwidth Bottleneck

Attention is where things get expensive, particularly at the memory level. During decode, each step requires:

1. Computing Q, K, V projections for the current token (three matrix-vector products of size d_model x d_model)
2. Appending the new K and V to the cache
3. Loading the entire KV cache for this request from HBM
4. Computing attention scores and the weighted sum over V

Step 3 is the bottleneck. The KV cache grows linearly with context length, so every additional cached token increases how much memory bandwidth the decode step consumes. At long contexts, a GPU can spend more time moving KV data from memory than doing any actual computation.

The Feed-Forward Network: Where Most FLOPs Go

The FFN in each transformer block typically has an intermediate dimension of 4x the hidden size. For a model with d_model = 4096, the FFN has two weight matrices of shape 4096 x 16384. Across a forward pass, the FFN accounts for roughly two-thirds of total FLOPs in a dense transformer.

During decode with a batch size of 1, these large matrix-vector multiplications (not matrix-matrix) are also memory-bandwidth-bound. You load the entire weight matrix to produce a single output vector. Throughput improves substantially with larger batches because you shift from matrix-vector to matrix-matrix operations, and GPU utilization recovers.

Residual Connections

Residual connections (the "skip" connections that add the layer input to its output) add two element-wise additions per block. Their computational cost is negligible compared to attention and FFN.

Attention Mechanisms and Their Inference Costs

The variant of attention a model uses has a large effect on KV cache size, which is frequently the binding constraint in serving.

Standard Multi-Head Attention (MHA)

In MHA, every attention head has its own Q, K, and V projections. A model with n_heads = 64 stores 64 K vectors and 64 V vectors per token per layer. This maximizes expressivity but also maximizes cache size.

Multi-Query Attention (MQA)

MQA (Shazeer, 2019) keeps multiple Q heads but uses a single shared K and V pair. The cache shrinks by a factor of n_heads. For a model with 64 heads, that's 64x less KV cache, at the cost of some model quality on harder tasks.

Grouped-Query Attention (GQA)

GQA is the standard in modern models. Query heads are organized into groups, with one K and V per group. Llama 3.1 70B uses 64 query heads and 8 KV heads, giving an 8x cache reduction versus MHA with minimal quality loss. Essentially all models released since 2024 use GQA. For a thorough comparison of these three variants, see Multi-Query and Grouped-Query Attention.

Multi-Head Latent Attention (MLA)

DeepSeek's MLA (used in DeepSeek-V2 and V3) takes a different approach: rather than reducing the number of KV heads, it projects K and V into a low-rank latent space before caching. The cache stores a compressed representation, which is expanded back to full K and V at attention time. The result is cache sizes smaller than GQA at equivalent quality. Details are in KV Cache Compression: MLA and Beyond.

How Context Length Quadratically Increases Attention Cost

In prefill, attention complexity is O(n^2) in sequence length -- each token attends to all previous tokens. A 128K-token input does not take twice as long to prefill as a 64K-token input; it takes approximately four times as long, because the number of attention pairs scales quadratically. Flash attention (see Flash Attention: Why Modern LLMs Run Faster With It) reorganizes the computation to be memory-efficient and avoids materializing the full attention matrix, but the quadratic FLOPs remain.

In decode, each step's attention cost grows linearly with the current context length, because you're attending over a growing cache.

Model Size, Parameters, and Memory Requirements

How to Calculate Model Memory Footprint from Parameter Count

The weight memory footprint is straightforward to estimate:

weight_bytes = num_parameters * bytes_per_param

Where bytes_per_param depends on precision:

• BF16 / FP16: 2 bytes
• FP8: 1 byte
• INT4 (with group scales): ~0.5 bytes (the scale factors add a small overhead)

Llama 3.1 70B has approximately 70.6 billion parameters. Weights only:

This determines whether a model fits on a single node. An H100 has 80GB of HBM, so Llama 70B at BF16 requires at least two GPUs just for the weights, before accounting for the KV cache and activations.

For quantization tradeoffs in more depth, see Quantization Explained: INT4, GGUF, GPTQ and What They Mean for Your Model.

KV Cache Memory -- Runtime Cost That Scales with Batch Size

The KV cache formula:

cache_bytes = 2 * n_layers * n_kv_heads * head_dim * seq_len * dtype_bytes * batch_size

The factor of 2 is for K and V. For Llama 3.1 70B (80 layers, 8 KV heads, head_dim 128, FP16) at 32K context, per request:

2 * 80 * 8 * 128 * 32,000 * 2 = 10.5 GB per request

This is the runtime cost that competes with your weight footprint for HBM. At 10.5 GB per request and 141 GB for weights in BF16 on an 8xH100 node (640 GB total), you have roughly 499 GB available for cache, which accommodates about 47 concurrent requests at 32K context. Move to FP8 KV cache and you double that to 94 requests.

At 128K context, the math changes:

2 * 80 * 8 * 128 * 128,000 * 2 = 42 GB per request

Now you can support roughly 11 concurrent requests on the same node at 32K context budget. Long-context serving is almost always a memory problem, not a compute problem.

Activation Memory

During inference (not training), activation memory is transient -- it's the intermediate tensors in the forward pass that exist only within a layer's computation. At inference batch sizes typical for serving (under a few hundred), activation memory is a small fraction of total GPU memory. It becomes a concern at very large batch sizes, where the hidden-state tensors across all tokens in the batch add up.

How Architecture Differences Affect Inference Speed

Dense vs Sparse (MoE) Models

In a Mixture of Experts model, each token is routed to a small subset of expert FFN layers (typically 2 out of 8 or 16). A model like Mixtral 8x7B has the FLOPs of a ~14B model per token but the parameter count of a ~47B model. This means:

• Lower compute cost per token (fewer FLOPs)
• Higher memory requirement (all experts must be loaded or accessible)
• Potential load imbalance issues under high concurrency

For throughput-bound workloads, MoE models can be very efficient: you get quality close to a larger dense model at a fraction of the FLOPs. For memory-bound workloads on small hardware, the full parameter footprint can be a constraint. See Mixture of Experts (MoE) Models: Why They're Dominating 2025 for a more detailed breakdown.

Context Length and Its Effect on Prefill Time

Prefill time grows quadratically with the length of the input, due to attention. For decode time, the cost per token grows linearly with the cached context length (you attend over more tokens per step). If your use case involves long system prompts repeated across requests, prefix caching (covered in Prefix Caching: Why Repeated Prompts Shouldn't Cost You Twice) avoids recomputing the shared prefix on every request.

Depth vs Width -- More Layers vs Larger Hidden Dimensions

Two models can have similar parameter counts with different shapes. A "deep" model has more layers with smaller hidden dimensions; a "wide" model has fewer layers with larger hidden dimensions.

For inference, depth creates a sequential dependency: each layer's output feeds the next, so layers cannot be parallelized across a single token's forward pass. More layers means more kernel launches and more pipeline bubbles in distributed serving. Width (larger hidden dim) increases FLOPs and memory bandwidth per layer but allows larger matrix operations that are more GPU-efficient.

In practice, depth and width interact with hardware in non-obvious ways. Shorter models (fewer layers) with the same parameter count can be faster on a single GPU because the sequential layer dependency chain is shorter, even if per-layer cost is higher.

Next-Generation Architectures Optimized for Inference

Mamba and State Space Models

Mamba (Gu and Dao, 2023) replaces attention with a selective state space model (SSM). Instead of attending over all past tokens, each token updates a fixed-size recurrent state. The inference cost per token is O(1) in context length, not O(n). This makes very long contexts computationally cheap -- no KV cache, no quadratic growth. The tradeoff is that SSMs are weaker at tasks requiring precise recall of specific information from earlier in the context, compared to attention. See Mamba and State Space Models: Inference Without Attention.

RWKV

RWKV reformulates attention as a linear recurrence, giving O(1) per-token inference cost like Mamba but using a different mathematical formulation. RWKV models can be efficient to serve because they require no KV cache and scale linearly in context length. See RWKV and Linear Attention: Recurrent Models as an Inference Shortcut.

Hybrid Models

Recent architectures combine attention layers with SSM or linear attention layers. The intuition is that full attention handles tasks requiring long-range recall, while linear layers handle most of the sequence modeling cheaply. This lets the model use a smaller KV cache (only for attention layers) while maintaining quality closer to full transformers than pure SSMs. Expect to see more hybrid architectures in high-throughput serving deployments over the next few years.

Practical Implications for Choosing a Model

How to Read a Model Card for Inference Efficiency

When evaluating a model's architecture, look for:

• Number of layers: 32 is typical for 7-8B; 80 for 70B. More layers = more sequential steps per token.
• Hidden dimension (d_model): 4096 for 7B, 8192 for 70B. Drives memory bandwidth per layer.
• Attention heads and KV heads: The ratio tells you which attention variant is used. Equal counts = MHA; fewer KV heads = GQA. Llama 3.1 8B: 32 Q heads, 8 KV heads. 70B: 64 Q heads, 8 KV heads.
• Intermediate (FFN) dimension: Usually 4x d_model, but some models use SwiGLU or other gated FFN variants that have their own size conventions.
• Expert count (for MoE): How many experts exist and how many are active per token. This determines effective FLOPs vs total parameter footprint.

Architecture Checklist Before Deployment

Before committing to a model for production:

• Calculate weight memory at your target precision. Does it fit on your hardware?
• Calculate KV cache per request at your expected context lengths. How many concurrent requests can you support?
• Check whether the model uses GQA. If it uses MHA, the cache footprint is much larger than you might expect.
• Check whether your serving framework supports the attention variant. Most do for GQA; MLA requires explicit support.
• For long-context use cases, check whether the model was actually trained at those lengths, not just "supports" them via rope scaling.

FAQ

Why do larger transformers take longer to run inference?

Larger models have more parameters, meaning more weight data must be read from HBM per token during decode. More layers also means more sequential operations per forward pass. Both effects increase latency. The FLOPs increase means compute also takes longer, though for single-token decode steps the memory bandwidth bottleneck usually dominates.

What is the KV cache and why does it use so much memory?

The KV cache stores the key and value vectors computed for each token in the context, so they don't need to be recomputed at every decode step. It grows with both context length and batch size. For a large model at long context, the cache can consume more memory than the model weights themselves. The formula above gives the exact calculation. See KV Cache in LLM Inference: How It Works and Why It Matters for a full treatment.

Is a model with fewer layers faster than one with more parameters?

Not necessarily, but depth is a real factor. Layers are sequential: you cannot overlap computation across them for a single token's forward pass. A 48-layer model and an 80-layer model with the same total parameter count will have the 48-layer model complete its forward pass faster per token, all else equal, because the sequential chain is shorter. In practice, width and depth are traded off by the model designers for quality reasons, and hardware-specific factors (kernel fusion, tensor parallelism layout) often matter more than raw layer count.

---

Architecture is the first constraint in any inference system. Getting familiar with the numbers -- weight bytes, cache bytes per request, FLOPs per token -- lets you reason about capacity, cost, and bottlenecks before you run a single benchmark. The rest is optimization on top of a foundation that's set when the model is trained.

If you want to run inference on models across the full size spectrum with ASIC-optimized throughput, you can try GeneralCompute's API directly.