For the last several years, the story of large language models has been one of brute force: to get more performance, you build a bigger, denser model. Every new state-of-the-art model seemed to follow this simple rule. But this approach has a crippling downside. In a dense model, every single input token requires the entire network—all of its billions or trillions of parameters—to light up and compute. The cost of training and inference scales directly with the model's size, pushing us toward a computational wall.

What if there was a more intelligent way to scale? What if a model could have a vast number of parameters, representing a huge repository of knowledge, but only use a small, relevant fraction of them for any given task? This is the core idea behind the Mixture-of-Experts (MoE) architecture, a paradigm that decouples model size from computational cost and is quickly becoming the standard for frontier models.

It's a seductive promise: the knowledge of a giant with the speed of a dwarf. But as always in engineering, there's no free lunch. Let's open the hood on how MoE really works.

• What is a Mixture-of-Experts Model?
• The MoE Layer: A Technical Breakdown
  • The Experts
  • The Gating Network (or Router)
  • Top-K Gating and Weighted Outputs
• The Secret Sauce: Load Balancing Loss
• The Payoff: Decoupling Parameters from FLOPs
• The Road Ahead: Sparsity is the New Black
  • Further Reading

What is a Mixture-of-Experts Model?

At its heart, a Mixture-of-Experts layer replaces a monolithic block of computation (typically a feed-forward network, or FFN, in a Transformer) with a collection of smaller, specialized sub-networks called "experts." Alongside these experts sits a "gating network" or "router"—a small neural network with a very important job: for each incoming token, it decides which experts are best suited to process it.

Think of it like a large consulting firm. A standard dense model is like a firm where every single consultant has to attend every meeting for every project—wildly inefficient. An MoE model is a firm with specialized departments (experts in finance, tech, law, etc.). When a client request comes in, a managing director (the gating network) doesn't waste everyone's time. They intelligently route the request to the one or two departments that have the relevant expertise.

This principle is called conditional computation. The computation is conditional on the input itself. Only the activated experts perform calculations, while the rest remain dormant.

The MoE Layer: A Technical Breakdown

An MoE layer is typically slotted into a Transformer architecture, replacing every FFN layer. So, instead of Attention -> FFN -> Output, the flow becomes Attention -> MoE -> Output.

The two key components are the experts and the router.

The Experts

The "experts" themselves are usually just standard feed-forward networks, identical in architecture to one another but with their own independent weights. Each one is trained to become specialized in processing certain types of information, though what these specializations represent is learned implicitly, not explicitly programmed.

The Gating Network (or Router)

This is where the magic happens. The gating network is a simple, learnable network that takes a token's embedding as input and outputs a probability distribution over all the available experts. In a typical implementation, this is a linear layer followed by a softmax function:

$P = \text{softmax}(x \cdot W_g)$

Where $x$ is the input token's embedding and $W_g$ is the learnable weight matrix of the gating network. The resulting vector $P$ contains the "votes" for how much each expert should contribute.

Top-K Gating and Weighted Outputs

Initially, one might think to just send the token to the single expert with the highest score (Top-1). However, research has shown that routing to a small number of experts—typically Top-2—leads to better performance and training stability.

In a Top-2 gating scheme, the gating network selects the two experts with the highest probability scores. The input token is then processed by both of these experts. Their final outputs are then combined via a weighted sum, where the weights are the normalized softmax scores from the gating network. This way, the model not only learns which experts to use but also how much to trust each one for a given token.

The Secret Sauce: Load Balancing Loss

So far, so good. But what's to stop the gating network from getting lazy? It could learn that a few experts are generally pretty good and start sending almost every token to them, leaving the other experts to gather digital dust. This would be a disaster. The "unpopular" experts would be undertrained and useless, and we'd be back to a few overworked parts of the network, defeating the purpose of having a large, distributed set of specialists.

To prevent this, MoE models introduce a critical component: a load balancing loss. This is an auxiliary loss function, added to the main model loss during training, that penalizes the uneven distribution of tokens to experts.

A common form of this loss is calculated based on the fraction of tokens dispatched to each expert ($f_i$) and the average probability for tokens dispatched to that expert ($P_i$). The total loss is:

$L*{\text{aux}} = \alpha \sum*{i=1}^{N} f_i \cdot P_i$

Here, $N$ is the number of experts and $\alpha$ is a scaling hyperparameter. By forcing the model to minimize this loss, we encourage the gating network to spread the load more evenly, ensuring all experts receive a rich and diverse set of training examples. It's the model's equivalent of a manager being told to ensure every department gets its fair share of work.

The Payoff: Decoupling Parameters from FLOPs

The true genius of MoE is that it decouples the number of model parameters from the floating-point operations (FLOPs) required for a forward pass.

This is the key to breaking the scaling laws of dense models. For example, the popular Mixtral 8x7B model has a name that describes its architecture: it contains 8 experts, each with approximately 7 billion parameters. This gives it a total parameter count of around 47B (not 56B, due to shared attention layers). However, for any given token, its Top-2 gating mechanism means it only ever uses the compute equivalent of a ~14B parameter model.

You get the vast knowledge and nuance of a nearly 50B-parameter model, but the inference speed of a much smaller one. This is why MoE models are not just a research curiosity but a practical breakthrough for building powerful yet efficient AI.

That said, if you think training a regular LLM is hard... try managing one of these. Training MoE models is a significant engineering challenge, requiring sophisticated parallelism strategies. You need data parallelism to split the data batches, but you also need expert parallelism, where the experts themselves are distributed across different GPUs, requiring high-speed communication between devices.

The Road Ahead: Sparsity is the New Black

While models like Mixtral have brought MoE into the limelight, the concept has been refined over years with key contributions like Google's GShard. Today, the challenges are shifting towards fine-tuning these sparse models effectively and exploring even more advanced forms of conditional computation.

As we continue to push the boundaries of what's possible with AI, the brute-force approach of building ever-larger dense models is hitting practical limits. Sparse architectures, with Mixture-of-Experts at the forefront, represent a more sustainable and computationally elegant path forward. They prove that in the future of AI, the smartest models may not be the ones that use all of their brain all the time, but the ones that know exactly which parts to use and when.

Further Reading

1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al. 2017)
2. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Lepikhin et al. 2020)
3. Mixtral of Experts (Mistral AI Blog)

Enjoyed this post? Subscribe to the Newsletter for more deep dives into ML infrastructure, interpretibility, and applied AI engineering or check out other posts at Deeper Thoughts