THE LEARNING PATH · Advanced

FlashAttention and the KV cache tame the GPU memory hierarchy

The real bottleneck is memory, not compute. Compare HBM and SRAM traffic, then estimate a model’s VRAM with a sizing tool.

9 min readVolume I · chapter 6, 9, 10 and 12

Memory before compute

Storing a 70-billion-parameter model in 16-bit precision already requires 70 × 2 bytes, about 140 GB. That is more than the 80 GB of an H100 GPU, and that is before adding activations, gradients, optimizer states or, at inference, the KV cache. On top of that, classic attention saturates bandwidth by multiplying round trips to the HBM (large but slow) to write N × N matrices there.

Three levers

FlashAttention reorganizes the exact computation by loading tiles into the chip’s tiny SRAM: the dot product and softmax are performed without ever writing the intermediate matrix to HBM, and activations are then recomputed on the fly during the backward pass. At inference, GQA and MQA (sharing the Key and Value heads) and quantization (INT8, 4-bit NF4) drastically reduce the VRAM footprint.

Analogies from Volume I: the warehouse, the shed and the scribe

FlashAttention: rather than shipping each part to a large faraway warehouse (the HBM), you work in whole crates directly in a small fast shed, right next to the crane (the SRAM).

KV cache and GQA: like a scribe who keeps past notes instead of rereading everything with each new word, the KV cache avoids recomputation. With GQA, several students (the Q heads) share a single textbook (K and V), which divides the cache memory.

Try it yourself

The first panel compares memory accesses; the second estimates the VRAM needed to serve a model with a batch of 8 sequences. The sizing tool gives an order of magnitude: it accounts for neither activations, nor fragmentation, nor the inference engine.

Interactive workshop

HBM ↔ SRAM flow and VRAM sizer

Toy model: it illustrates the mechanism with small examples and does not measure a real model.

Key takeaways

  • FlashAttention computes exactly the same mathematical formula, but removes the input/output bottleneck between HBM and SRAM.
  • Moving from MHA to GQA (adopted by LLaMA 2 and 3) divides the KV cache size by a factor of 4 to 8, which makes it possible to serve more concurrent contexts.
  • 4-bit quantization (NF4, QLoRA) shrinks the weights by a factor of 4 compared with FP16 and democratizes local use.

CHECK

Check your understanding

To validate

Why is FlashAttention faster than standard attention, even though it sometimes performs more mathematical operations (recomputation during the backward pass)?