THE LEARNING PATH · Intermediate
Mixture of Experts specializes computation without weighing down inference
A router activates only a few experts per token. Simulate Top-K routing and collapse, and tell total parameters from active ones.
Dense versus sparse
In a dense Transformer, every generated word activates 100% of the model’s billions of parameters. The MoE (Mixture of Experts) architecture replaces the feed-forward layers, which hold about two thirds of the weights, with a set of independent experts driven by a router.
For each token, the router selects only the K best experts (sparse top-K, often 1 or 2). Mixtral 8×7B thus offers the knowledge of about 46.7 billion parameters while activating only about 13 billion per token; DeepSeek-V3 activates 37 billion parameters out of 671 billion in total.
ŷ = Σ G(x)ᵢ · Eᵢ(x) for i in the Top-K
L_aux ∝ Σ [ (P(G(x)ᵢ) − 1/N)² + (fᵢ − 1/N)² ]
Analogy from Volume I: the general hospital
A dense model is a general practitioner trying to solve everything alone. An MoE is a large hospital: at reception, a triage nurse (the router) examines each patient (each token) and directs them to the two relevant specialists.
Without an auxiliary loss during training, the router becomes lazy: a “superstar” expert captures all the traffic and the others atrophy. This is routing collapse.
Try it yourself
Select a token and observe which experts activate. Then switch to “no auxiliary loss” mode to trigger collapse. The routing scores are hand-written examples, and the specialty names are a simplification: in a real model, experts do not sort themselves so neatly.
Interactive workshop
Top-K routing and collapse simulator
Toy model: it illustrates the mechanism with small examples and does not measure a real model.
Key takeaways
- An MoE model separates knowledge capacity (total parameters) from the compute cost per token (active parameters).
- The VRAM footprint remains dictated by the total parameters: all the experts must stay ready in memory.
- Comparing a “47B” MoE to a dense model without distinguishing total from active parameters is misleading.
CHECK
Check your understanding
What concrete advantage and what hardware constraint characterize a sparse MoE model like Mixtral 8×7B at inference?