THE LEARNING PATH · Intermediate
RoPE encodes relative distance through a geometric rotation
Rather than adding a position, RoPE rotates Queries and Keys. Test on a dial why only the gap matters.
Without order, a bag of words
Without any indication of order, attention sees a bag of words: “John eats the lion” and “The lion eats John” become indistinguishable. The first models used learned position embeddings, frozen at the maximum length seen during training: beyond the 512th token, the model was lost.
Rotation as position
Rotary position encoding (RoPE, adopted by LLaMA, Mistral, Qwen and DeepSeek) removes this barrier. Rather than adding a position vector to the input, a rotation by angle m·θ is applied directly to the Query and Key vectors. The dot product then depends only on the relative distance (n − m).
⟨R(mθ)·q, R(nθ)·k⟩ = ⟨q, R((n − m)θ)·k⟩ ∝ cos((n − m)θ)
Analogy from Volume I: the hands on a dial
Imagine each vector as a hand on a dial. At position m, the hand is turned by an angle proportional to m.
If two words 3 positions apart appear at indices 2 and 5, or much further along, at indices 10,002 and 10,005, the angular gap between their two hands remains rigorously identical. The model therefore generalizes naturally to long contexts.
Try it yourself
Drag the positions m and n, then use “Shift together”: both hands turn, but the arc separating them and the cosine do not move. The dial shows a single pair of dimensions; a real model combines dozens of them with different base angles.
Interactive workshop
RoPE trigonometric dial
Toy model: it illustrates the mechanism with small examples and does not measure a real model.
Key takeaways
- RoPE acts by rotating the Q and K vectors before the dot product, without modifying the Value vectors V.
- Depending exclusively on the relative distance (n − m) preserves syntactic patterns even across tens of thousands of tokens.
- In practice, this avoids a collapse in coherence as conversations grow longer.
CHECK
Check your understanding
If two words are simultaneously shifted by +500 positions toward the end of a long document under RoPE, what happens to the relative angle between their Q and K vectors (for the same semantic vector)?