What is a mixture-of-experts model?

A mixture-of-experts (MoE) model is a conditional-computation architecture in which a learned gating network decides, per input, which of several parallel expertsub-networks should process that input. The idea predates deep learning — Jacobs, Jordan, Nowlan, and Hinton introduced “Adaptive Mixtures of Local Experts” in Neural Computationin 1991 — but it became the dominant scaling strategy for large language models after Shazeer et al.’s “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” (arXiv:1701.06538, 2017), which showed an LSTM with 137 billion parameters could be trained on TPU clusters by activating only ~2 of 2048 experts per token.

The mechanism is simple. A standard Transformer feed-forward layer is replaced by N independent feed-forward “experts,” typically 8 to 256 of them. A small gating function (usually a learned linear projection followed by softmax) scores the experts for each token, and a top-k router selects the k highest-scoring experts (commonly k=1 or k=2). Only those k experts run; their outputs are weighted by the gate scores and summed. Because k is much smaller than N, the active parameter count per token is far below the total parameter count. Mistral AI’s Mixtral 8x7B (arXiv:2401.04088, January 2024), for example, has 46.7B total parameters but only 12.9B active per token because k=2 of 8 experts fire per layer.

This sparsity is the entire point. Dense models pay for every parameter on every token; MoE models pay only for the experts the router picks, so they trade some routing overhead and memory footprint for a dramatically lower FLOPs-per-token cost. Google’s GShard (arXiv:2006.16668, 2020) scaled this to a 600B-parameter translation model, and Switch Transformer (arXiv:2101.03961, 2021) showed that even k=1 routing — the simplest possible MoE — could pretrain 7x faster than a dense T5 baseline at matched compute.

The hard problems in MoE are load balancing and routing instability. If the gate sends most tokens to a handful of popular experts, the others starve and never learn; if the routing decision is too noisy, training diverges. Switch Transformer introduced an auxiliary load-balancing loss that penalizes uneven expert utilization. Expert Choice Routing (arXiv:2202.09368, 2022) inverted the formulation — experts pick tokens rather than tokens picking experts — which guarantees balance by construction. DeepSeek-V3 (arXiv:2412.19437, December 2024) pushed this further with an auxiliary-loss-free balancing scheme and 671B total / 37B active parameters across 256 routed experts plus 1 shared expert per layer.

MoE has costs too. Total parameter count drives memory and inter-GPU communication, not just compute, so MoE models need high-bandwidth interconnects (NVLink, InfiniBand) and expert parallelism— sharding experts across devices — to train and serve efficiently. Mixtral 8x7B requires roughly 90GB of VRAM in bf16 despite its 13B active footprint. The router is also a single point of failure: a poorly trained gate produces “dead experts” that never fire, wasting capacity.

As of 2025–2026, MoE is the default for frontier open-weight models. Mixtral 8x22B, DeepSeek-V3, Qwen3-235B-A22B, Llama 4 Maverick (17B active / 400B total, 128 experts), and Grok-1 (314B total, 2 of 8 experts per token) are all sparse MoE Transformers. Closed models are widely reported to use MoE as well — GPT-4 was described as a 16-expert MoE in multiple credible analyses, though OpenAI has not officially confirmed the architecture.