What Is Multi-head Latent Attention—and How Does It Shrink AI's Memory?

Hello, and welcome to In Simple Terms with Satish.

Have you ever monitored a local chatbot's GPU-memory meter while a conversation grows? Start with a short prompt, then paste a long report and continue chatting. The meter can climb even though the model's weights have not changed.

So what exactly is accumulating?

The answer begins with a key-value cache, or KV cache. It saves the model from recalculating the full history after every generated word, but the cache itself grows with that history.

During attention, the newest token produces a query: roughly, what information am I looking for? Earlier tokens have keys that help determine where to look and values containing information that can be retrieved. Once a past token's keys and values have been calculated, the system stores them. When the next token is generated, it reuses those stored representations.

Put numbers on that growth with an illustrative configuration—not a claim about every model. Take 32 layers, eight KV heads, a head dimension of 128, separate keys and values, and two bytes per value. Multiply 32 by 8 by 128 by 2 by 2: that is 131,072 bytes, or 128 kibibytes, for each cached token. At 16,000 tokens, one sequence needs about 1.95 gibibytes—roughly two gibibytes—before model weights, activations, temporary buffers, or other users.

Imagine a meeting where every participant keeps a full copy of the notes. That resembles standard Multi-Head Attention, or MHA: each attention head stores its own key and value for every token. Grouped-Query Attention, or GQA, lets groups of heads share fewer key-value sets.

MLA changes the record itself. Picture one compact notebook entry per statement, shared across participants, plus a timestamp. The entry maps to a learned low-rank key-value latent; the timestamp maps to the separate positional component; the participants map to attention heads.

When a head needs history, learned projections interpret the shared latent. Rather than keeping every expanded per-head key and value, the system caches the compact latent and positional part.

Here is the boundary: the latent is learned numerical state, not a readable summary, and every prior token still has an entry. MLA changes representation; it does not choose important words or guarantee correct reasoning.

There is one complication: word order.

Transformers often use Rotary Position Embedding, or RoPE, to encode where a token appears. Its rotation depends on position. If that position-dependent operation is mixed directly into the compressed content path, it interferes with absorbing fixed projection weights efficiently.

MLA addresses this with decoupled RoPE. It keeps a small positional key component separate from the compressed content latent. Attention can then combine content similarity with positional information while caching the compact latent plus the smaller positional component.

Why does this matter in production?

For one short prompt, cache memory may not dominate. Across long contexts or many concurrent users, reducing the bytes stored per token can leave room for more requests, larger batches, or longer sequences. It may also reduce memory traffic during decoding, where moving cached data can be a bottleneck.

But MLA is not free compression.

The model must be designed and trained for this representation, or carefully converted and adapted. If the latent bottleneck is too narrow, useful information may be lost. Projection work still happens, optimized kernels matter, and the best latency result depends on the hardware and workload. The cache becomes smaller; it does not disappear.

MLA also does not give a model perfect memory. It preserves numerical attention state more compactly. Whether the model notices an important fact from fifty pages earlier remains a question of training, attention behavior, and evaluation—not cache capacity alone.

The essential contrast is this: MHA stores separate keys and values for many heads; GQA shares them across groups; MLA stores a compressed latent content representation and a small positional component.

So when a model supports a very long context, ask not only how many tokens fit. Ask how much memory each cached token requires, what attention architecture is used, and whether real retrieval quality survives at that length.

If this helped you understand the topic in simple terms, follow or subscribe to In Simple Terms with Satish, and share this episode with one friend who likes technology. I will see you in the next episode.

Download plain-text transcript