THE LEARNING PATH · Intermediate

Mixture of Experts specializes computation without weighing down inference

A router activates only a few experts per token. Simulate Top-K routing and collapse, and tell total parameters from active ones.

8 min readVolume I · chapter 7

Dense versus sparse

In a dense Transformer, every generated word activates 100% of the model’s billions of parameters. The MoE (Mixture of Experts) architecture replaces the feed-forward layers, which hold about two thirds of the weights, with a set of independent experts driven by a router.

For each token, the router selects only the K best experts (sparse top-K, often 1 or 2). Mixtral 8×7B thus offers the knowledge of about 46.7 billion parameters while activating only about 13 billion per token; DeepSeek-V3 activates 37 billion parameters out of 671 billion in total.

ŷ = Σ G(x)ᵢ · Eᵢ(x)   for i in the Top-K
L_aux ∝ Σ [ (P(G(x)ᵢ) − 1/N)² + (fᵢ − 1/N)² ]

Analogy from Volume I: the general hospital

A dense model is a general practitioner trying to solve everything alone. An MoE is a large hospital: at reception, a triage nurse (the router) examines each patient (each token) and directs them to the two relevant specialists.

Without an auxiliary loss during training, the router becomes lazy: a “superstar” expert captures all the traffic and the others atrophy. This is routing collapse.

Try it yourself

Select a token and observe which experts activate. Then switch to “no auxiliary loss” mode to trigger collapse. The routing scores are hand-written examples, and the specialty names are a simplification: in a real model, experts do not sort themselves so neatly.

Interactive workshop

Top-K routing and collapse simulator

Toy model: it illustrates the mechanism with small examples and does not measure a real model.

Key takeaways

  • An MoE model separates knowledge capacity (total parameters) from the compute cost per token (active parameters).
  • The VRAM footprint remains dictated by the total parameters: all the experts must stay ready in memory.
  • Comparing a “47B” MoE to a dense model without distinguishing total from active parameters is misleading.

CHECK

Check your understanding

To validate

What concrete advantage and what hardware constraint characterize a sparse MoE model like Mixtral 8×7B at inference?