Mixture of Experts (MoE) Models: Why They're Dominating 2025

If you look at the models that have dominated benchmark leaderboards over the past two years, most of them are Mixture of Experts (MoE) architectures. Mixtral, DeepSeek V3, Grok-1, Llama 4 Maverick -- all MoE. The architecture has become the default for anyone building at the frontier, and there's a clear reason why: MoE lets you scale model capacity without scaling compute proportionally.

This post explains how MoE works, what the gating and routing mechanics look like in practice, and where the hard problems actually are. If you've seen the term and want to understand what's really happening under the hood, this covers it.

The core idea: conditional computation

A standard transformer processes every token with the same set of parameters. Every token passes through the same attention heads and the same feed-forward network (FFN). This means scaling quality requires scaling compute, roughly linearly.

MoE breaks that constraint by making parts of the network conditional. Instead of one shared FFN, an MoE layer has N separate FFN networks called "experts," plus a small router network that decides which experts each token uses. For any given token, only k of the N experts are active. The others don't run. You get a model with far more total parameters, but each forward pass only touches a fraction of them.

This is sometimes called "sparse" computation, in contrast to the "dense" computation of a standard transformer. The sparsity is what creates the efficiency: you can make the model much larger without making it proportionally slower.

How the router and gating work

The router is usually a simple single linear layer. It takes the token's hidden state as input and outputs a logit score for each expert. These scores determine which experts handle the token.

The selection process looks like this:

gate_logits = router(hidden_state)  # [num_experts]
top_k_indices = torch.topk(gate_logits, k=2).indices
top_k_weights = F.softmax(gate_logits[top_k_indices], dim=-1)

# output is a weighted sum of the chosen experts
output = sum(
    top_k_weights[i] * experts[top_k_indices[i]](hidden_state)
    for i in range(k)
)

The router runs on every token, at every MoE layer, in every forward pass. The expert outputs are combined using the gating weights (the softmax scores), so the contribution of each expert is proportional to how strongly the router scores it for that token.

Most practical MoE models use k=2, meaning two experts are active per token per layer. Some recent fine-grained designs use higher k with smaller experts, but the principle is the same: a small number of experts activate, the rest are skipped.

The attention layers in an MoE model are almost always kept dense -- every token still goes through the same attention computation. The sparsity applies only to the FFN portion. In modern transformers, the FFN accounts for two-thirds or more of the total parameter count, so this is where most of the savings come from.

Why this produces better models

The interesting result, confirmed across many experiments, is that MoE models consistently outperform dense models of equivalent active-parameter count. A model with 8B active parameters arranged as MoE will often beat a 8B dense model on standard benchmarks.

The intuition is that different experts can specialize. Over training, the router learns to direct certain types of tokens or patterns to experts that have become good at handling them. One expert might specialize in code, another in mathematical reasoning, another in natural language. The model develops more diverse internal representations than a dense model of the same active size.

This specialization is emergent, not explicit. You don't assign experts to domains. The routing and the expert weights are learned jointly, and specialization arises from gradient flow. There's evidence of it in post-hoc analyses (you can sometimes identify expert specialization by measuring what input distributions each expert activates on), but it's not guaranteed to be clean or interpretable.

The practical implication is that you can match the quality of a dense 70B model with an MoE model that has 70B active parameters at most -- but you can spread those "total" parameters across many more experts, increasing total capacity significantly. DeepSeek V3 pushes this hard: 671B total parameters, 37B active per token. It consistently matches or beats models with much larger active-parameter counts.

The models that made MoE mainstream

Mixtral 8x7B (Mistral, 2023) was the first widely adopted open MoE model. It has 8 experts per layer and activates 2, giving it 46.7B total parameters with 12.9B active per token. It outperformed Llama 2 70B on most benchmarks while being much cheaper to serve at comparable quality. This demonstrated that MoE was practical to deploy and not just a research curiosity.

Mixtral 8x22B followed in 2024, scaling to 141B total parameters with 39B active. It pushed the quality ceiling further while keeping active-parameter compute manageable.

Grok-1 (xAI, 2024) was released as an open-weights model with 314B total parameters and 86B active per token. It uses 8 active experts out of 64, making it one of the largest open-weights MoE models at the time of release.

DeepSeek V3 is currently the most influential MoE model in production. 671B total parameters, 37B active, with 256 routed experts plus 2 shared experts per layer. The fine-grained routing (many small experts rather than few large ones) combined with hardware-aware training let it match or beat GPT-4-class models on most benchmarks while being significantly cheaper to train. DeepSeek's architecture paper was heavily studied for the engineering choices it documents.

Llama 4 adopted MoE across its Maverick and Scout variants. Maverick has 400B total parameters with 17B active per token. This marked MoE's full arrival into Meta's core model family, which had been dense through Llama 3.

The pattern across all these models is similar: MoE was chosen specifically to increase total capacity beyond what was practical with dense architectures at the available compute budget.

Training challenges

MoE models are harder to train than dense models, and the main difficulty is load balancing.

If the router learns to prefer a small subset of experts for most tokens, those experts receive more gradients and improve faster. The other experts get fewer gradients, improve slower, and the router has even less reason to route to them. This is a collapse mode called "expert collapse" or "routing collapse," and it's a real failure case in MoE training.

The standard solution is an auxiliary load-balancing loss added to the training objective:

# fraction of tokens routed to each expert
router_probs = softmax(gate_logits, dim=-1)  # [batch, num_experts]
fraction_per_expert = router_probs.mean(dim=0)

# target: uniform distribution (1/num_experts per expert)
load_balance_loss = num_experts * (fraction_per_expert * fraction_per_expert).sum()
total_loss = task_loss + alpha * load_balance_loss

The coefficient alpha controls how strongly you enforce balance. Too low and experts collapse; too high and the routing loses expressivity because the router is forced to distribute uniformly regardless of input, defeating the purpose of specialization.

Getting this balance right is one of the main hyperparameter challenges in MoE training. DeepSeek V3 used an expert-level bias term that adjusts routing probabilities based on recent expert utilization history, which allowed them to maintain balance without aggressively penalizing the task loss.

Training stability is also harder. The sparse routing creates high variance in gradient estimates, particularly for the router. Careful initialization, gradient clipping, and monitoring of routing statistics (which experts are active, how uniform the distribution is across the batch) are all more important than in dense training.

Inference and load balancing challenges

At inference time, load balancing takes a different form. During training you can compute the auxiliary loss and push the router toward uniformity. At inference time, you just have the trained router making decisions for whatever input comes in. If your workload happens to trigger many tokens that all route to the same expert, some GPUs sit idle while one is overwhelmed.

For large MoE models served with expert parallelism (each GPU holds a subset of experts), this shows up as throughput instability. The all-to-all communication that ships tokens to their assigned expert GPUs becomes uneven, and the latency of a batch is determined by the slowest expert, not the average.

Production serving addresses this several ways:

Capacity capping: each expert accepts only up to a capacity factor times the average load. Tokens that exceed the cap get dropped to their second-choice expert or skipped entirely. This caps worst-case latency at the cost of a small quality hit.
Auxiliary routing tables: some serving stacks maintain a soft router that can redirect traffic away from overloaded experts in real time. This adds complexity but can smooth out load spikes.
Batching strategy: with large enough batches and diverse enough inputs, the law of large numbers tends to even out routing. Single-request serving at low batch sizes sees more variance; high-throughput batch serving sees more natural balance.

In practice, the load balancing issue is real but manageable. The serving frameworks (vLLM, SGLang, TensorRT-LLM) have all invested in efficient dispatch kernels that minimize the overhead from irregular routing patterns. The main signal to watch is per-expert token counts within a batch -- if they're highly non-uniform, you'll see throughput well below theoretical peak.

Memory vs. compute trade-off

One thing that catches people off guard: MoE models are not smaller in memory than their dense equivalents. All the experts have to be loaded into GPU memory, even if only 2 of 256 are active for any given token. DeepSeek V3 at 671B parameters requires roughly the same GPU memory as a 671B dense model would.

The economics work because the compute savings are large. A 671B MoE model with 37B active parameters does about 37B/671B = 5.5% of the FLOPs per token of a theoretical 671B dense model. The memory footprint stays large; the compute footprint drops dramatically. For throughput-bound workloads -- where you're processing many requests and the bottleneck is GPU compute rather than memory bandwidth -- this is a good trade.

For latency-bound workloads at very low batch sizes, the trade is less attractive. You're paying full memory bandwidth cost on every decode step regardless of which experts activate, because the attention layers are still dense. The per-token compute savings help, but they don't fully offset the memory bandwidth cost of the large model.

This is why you'll see MoE models favored for high-throughput serving scenarios and why dense models remain competitive at small scale. A 7B dense model outpaces a 671B MoE model on a single GPU for interactive, single-user latency because it's simply much smaller and faster to move through memory.

Where MoE fits in the architecture landscape

The practical picture in 2025 is that MoE has become the default for frontier-scale models. If you're training a model with more than a few hundred billion parameters and you want to be compute-efficient, MoE is the expected choice. Dense models at that scale are too expensive to train and too slow to serve.

For smaller models, dense is still dominant. The routing overhead, training complexity, and memory requirements of MoE don't pay off below roughly 30B total parameters. The sweet spot for MoE is models where you want 70B+ quality but can't afford to activate 70B parameters per token.

The interesting design space is in the middle: models in the 30B to 150B total parameter range, where MoE and dense are both viable and the tradeoffs are genuine choices. Here the routing granularity (few large experts vs. many small experts), the activation ratio, and the serving infrastructure all matter for how the model performs in production.

Starting with MoE models

If you want to use MoE models without setting up the infrastructure yourself, General Compute runs DeepSeek V3 and Llama 4 Maverick on production inference hardware with expert parallelism and the dispatch kernels already in place. The API is OpenAI-compatible; switching from a dense model is a one-line change.

from openai import OpenAI

client = OpenAI(
    api_key="your-api-key",
    base_url="https://api.generalcompute.com/v1"
)

response = client.chat.completions.create(
    model="deepseek-v3",  # 671B MoE, 37B active
    messages=[{"role": "user", "content": "Explain the gating mechanism in MoE models."}]
)
print(response.choices[0].message.content)

For deeper coverage of how MoE behaves specifically at inference time -- including the all-to-all communication, per-layer dispatch, and expert parallelism topology -- see Mixture of Experts at Inference Time.

You can get started at generalcompute.com.