THE LEARNING PATH · Beginner

Splitting text into tokens determines cost and latency

Before any matrix multiplication, text becomes a sequence of identifiers. Try a BPE tokenizer and see the effect on cost.

7 min readVolume I · chapter 2 and 3

The token: the unit the model actually reads

Before the slightest matrix multiplication, a language model converts the continuous stream of text into a discrete sequence of integer identifiers: the tokens. This choice of unit is a permanent trade-off between vocabulary size and sequence length.

Word-level splitting fails on shared roots (« bear » and « bears ») and stumbles on any term missing from the dictionary, known as out-of-vocabulary. Conversely, character-level splitting eliminates unknown words but stretches sequences enormously. The current standard, BPE (Byte-Pair Encoding) or WordPiece, statistically merges the most frequent pairs to obtain reusable subwords.

An invisible but very real cost

In production, this discrete step directly determines the bill. A tokenizer trained mostly on English fragments French more: for the same sentence, the number of tokens goes up, and so do billing, latency and memory consumption.

Analogy from Volume I: the digital score

Imagine tokenization as turning a flowing melody into sheet music. The continuous text is cut into individual notes, the tokens, which the instrument can then play.

Two sentences of the same length in characters can produce a very different number of “notes”. The KV cache cost then varies linearly with that number, and the initial attention computation varies quadratically.

Try it yourself

In the workshop below, compare the three granularities on the provided examples or on your own sentence. Pay particular attention to the relative cost column: it is what links a splitting decision to a bill.

Interactive workshop

Token splitter and cost impact

Toy model: it illustrates the mechanism with small examples and does not measure a real model.

Key takeaways

  • Modern models (GPT, LLaMA, Mistral) split text into subwords to balance vocabulary size against sequence length.
  • APIs bill by the number of tokens, never by the number of words or characters.
  • A repeated or unusual word (« hellooooooo ») breaks into several tokens and increases latency.

CHECK

Check your understanding

To validate

Why does a BPE tokenizer trained mostly on English cost more to use on French text of the same length in characters?