THE LEARNING PATH · Beginner

The geometry of latent space turns meaning into vectors

Embeddings place words in a space where relationships become translations. Manipulate them in a 3D observatory.

7 min readVolume I · chapter 3 and 11

From one-hot to dense vectors

A simple one-hot encoding assigns each token a giant vector, all mutually orthogonal. Mathematically, “dog” is as far from “wolf” as it is from “table”. To capture semantics, networks learn dense vectors: the embeddings, stored in a weight matrix of size vocabulary × dimension.

Introduced by Word2Vec through a proxy task (guessing a word from its context), this learning organizes concepts in space. Logical relationships become vector translations, a property that persists and grows stronger in the hidden states of modern LLMs (this is known as linear representation).

v(result) = v(King) − v(Man) + v(Woman) ≈ v(Queen)

Analogy from Volume I: the map of meaning

Picture each embedding as GPS coordinates on an immense conceptual map. On this map, “Paris” and “France” have the same displacement vector as “Rome” and “Italy”.

Unlike Word2Vec’s static vectors, the attention layers of Transformers produce contextual embeddings: “bank” moves depending on whether it sits next to “money” or “river”.

Try it yourself

The observatory below projects thirteen words into a 3D space. The coordinates are fictitious and chosen to keep the example readable: a real space has hundreds of dimensions. Drag the translation from 0 to 100% and watch which word becomes the closest.

Interactive workshop

Vector observatory and the arithmetic of meaning

Toy model: it illustrates the mechanism with small examples and does not measure a real model.

Key takeaways

  • The embedding matrix is part of the model’s parameters (vocabulary size × dimension).
  • A vector’s direction carries semantic information; its angular proximity (the cosine) underpins vector search and RAG.
  • Two different models have incompatible vector spaces: you never compare their embeddings directly.

CHECK

Check your understanding

To validate

In the “proxy task” technique used by Word2Vec (CBOW or Skip-gram), what is the true objective of the training?