THE LEARNING PATH · Intermediate
Multi-head attention links each word to its context in parallel
Queries, keys and values: see how each token queries the others, and why we divide by √d_k before the softmax.
Breaking sequential reading
Recurrent networks (RNNs) read text word by word and suffered from amnesia on long sentences (the vanishing gradient), while underusing GPUs. The Transformer breaks this sequentiality thanks to self-attention: each token talks simultaneously with all the others, in a series of matrix operations.
Query, Key, Value
For each attention head, three linear projections give each token a Query (Q), a Key (K) and a Value (V). Dividing by √d_k keeps the dot products from growing too large and choking the softmax gradients. Stacking several heads in parallel makes it possible to track grammar, polysemy and coreferences all at once.
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
Analogy from Volume I: the library search
The Query is your mental question (“I’m looking for a book about planets”). The Key is the label on the spine of the books (“Astronomy”, “Cooking”, “History”). The Value is the content of the book that you extract when the Query and the Key match.
With multi-head attention, a polysemous word like “bank” is read in several ways: one head links it to “The” (syntax), another to “money” (financial sense), a third to “river” (natural sense).
Try it yourself
Choose a sentence, a head and a Query word. Then switch to “no division” mode: with d_k = 16 in this example, the scores are multiplied by 4 and the softmax concentrates on a single peak. The scores used are small hand-written examples, not those of a real model.
Interactive workshop
Q·K·V inspector and softmax saturation
Toy model: it illustrates the mechanism with small examples and does not measure a real model.
Key takeaways
- Attention weights form a probability distribution (summing to 1 per row) that dictates the proportion of each Value that is aggregated.
- In a generative decoder (GPT, LLaMA), a causal mask prevents each token from looking at future words.
- Multi-head attention learns distinct subspaces (syntax, meaning, coreferences) without losing GPU parallelism.
CHECK
Check your understanding
Why do we divide the dot product Q·Kᵀ by √d_k before applying the softmax function?