# Why is FlashAttention faster and lighter on memory?

> Standard attention is slow because the N×N score matrix is written to HBM, not because the GPU lacks FLOPs. FlashAttention tiles that work into SRAM and still computes exact softmax.

Canonical: https://zzszmyf.github.io/en/flashattention/


## Answer

Standard attention is slow because the N×N score matrix is read and written through HBM, not because the chip is short of FLOPs. FlashAttention never materializes that matrix. It cuts the work into tiles that fit in on-chip SRAM, runs online softmax so the blocked result is still exact, and recomputes in the backward pass instead of storing the big matrix.

## What to remember

Tiling answers the memory question. Online softmax answers the “softmax needs the whole row” question. Recompute answers the training-memory question. The algorithm is exact; the speedup is fewer HBM round trips. The Chinese note walks the IO bound, a hand calculation of the online update, and how FA1, FA2, and FA3 line up with GPU generations.

## When this is the wrong lever

Decode of one token is often bandwidth-bound on the weights and the KV cache, not on an N×N score matrix. Cutting KV heads (MQA, GQA, MLA) or reusing a prefix is a different problem.