- 孟一凡（zzszmyf）· LLM 推理优化与 AI Agent 技术笔记/
- English summaries/
- MQA, GQA, or MLA: what should you cache per token?/
MQA, GQA, or MLA: what should you cache per token?
目录
Answer #
The question is what you store per token, not how you compute the score matrix. Multi-head attention keeps a K and a V for every head. Multi-query attention keeps one K/V pair shared by all heads. Grouped-query attention sits between them: several heads share one pair. MLA, as used by DeepSeek-V2 in the Chinese note, compresses K and V together into a low-rank latent.
GQA is the usual compromise when you can up-train from a multi-head checkpoint. MLA’s extra catch is RoPE: the note derives why a plain low-rank latent fights rotary position and why decoupled RoPE is the workaround. The chapter’s 70B and DeepSeek-V2 memory tables are the place to copy a formula, not this page.
What this does not fix #
Fewer KV heads do not remove the HBM traffic of a long prefill score matrix. That is FlashAttention. They also do not reuse a prefix across requests. That is RadixAttention.