THE LEARNING PATH · Beginner
The geometry of latent space turns meaning into vectors
Embeddings place words in a space where relationships become translations. Manipulate them in a 3D observatory.
From one-hot to dense vectors
A simple one-hot encoding assigns each token a giant vector, all mutually orthogonal. Mathematically, “dog” is as far from “wolf” as it is from “table”. To capture semantics, networks learn dense vectors: the embeddings, stored in a weight matrix of size vocabulary × dimension.
Introduced by Word2Vec through a proxy task (guessing a word from its context), this learning organizes concepts in space. Logical relationships become vector translations, a property that persists and grows stronger in the hidden states of modern LLMs (this is known as linear representation).
v(result) = v(King) − v(Man) + v(Woman) ≈ v(Queen)
Analogy from Volume I: the map of meaning
Picture each embedding as GPS coordinates on an immense conceptual map. On this map, “Paris” and “France” have the same displacement vector as “Rome” and “Italy”.
Unlike Word2Vec’s static vectors, the attention layers of Transformers produce contextual embeddings: “bank” moves depending on whether it sits next to “money” or “river”.
Try it yourself
The observatory below projects thirteen words into a 3D space. The coordinates are fictitious and chosen to keep the example readable: a real space has hundreds of dimensions. Drag the translation from 0 to 100% and watch which word becomes the closest.
Interactive workshop
Vector observatory and the arithmetic of meaning
Toy model: it illustrates the mechanism with small examples and does not measure a real model.
Key takeaways
- The embedding matrix is part of the model’s parameters (vocabulary size × dimension).
- A vector’s direction carries semantic information; its angular proximity (the cosine) underpins vector search and RAG.
- Two different models have incompatible vector spaces: you never compare their embeddings directly.
CHECK
Check your understanding
In the “proxy task” technique used by Word2Vec (CBOW or Skip-gram), what is the true objective of the training?