KV cache explained: how it works and what it costs

The KV cache is the inference engine's saved attention state: one key and one value per key/value head, per token, per layer, held in GPU memory for the life of a request. Without it, every new token redoes the attention work for the entire prefix. A 2,000-token reply to an 8,000-token prompt would run the model over 8,000 to 10,000 tokens, 2,000 times. With it, each step processes one token, and decode turns from a compute problem into a memory problem.

This page covers how the cache works, how to size it, and what shrinks it. For the surrounding mechanics, see how large language models work.

How the KV cache works during generation

The cache optimizes the loop described in how an LLM generates text one token at a time. That loop has two phases.

In prefill, the model processes the whole prompt in parallel and writes a key (K) and value (V) for every prompt token in every layer. In decode, it generates one token per forward pass. It computes the query (Q), K and V for the new token only, appends K and V to the cache, and attends that one query over everything stored. Per layer, for B requests with T tokens so far:

prefill  K, V for the prompt    [B, kv_heads, T,   head_dim]  -> write to cache
decode   Q for the new token    [B, q_heads,  1,   head_dim]
         K, V for the new token [B, kv_heads, 1,   head_dim]  -> append
         cached K, V            [B, kv_heads, T+1, head_dim]
         weights = softmax(QKᵀ / √head_dim)
                                [B, q_heads,  1,   T+1]
         output  = weights · V  [B, q_heads,  1,   head_dim]

Where q_heads exceeds kv_heads, each K/V head is shared by a group of q_heads / kv_heads query heads.

Only keys and values are cached because past queries are never used again. Attention is causal, so the newest token's query is the only one with work left. Every stored K and V gets read again at every step.

The trade is compute for memory. Attention work per decode step drops from quadratic in sequence length to linear. In exchange, each step reads the weights and the whole cache out of GPU memory to produce one token, which makes decode memory-bandwidth-bound. Most LLM inference tuning follows from that.

How much memory the KV cache uses

The size comes from three numbers in the model's config.json and one serving choice, precision:

bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
total bytes     = bytes per token × sequence length × batch size

The 2 is one K plus one V. num_layers is num_hidden_layers, and num_kv_heads is num_key_value_heads. head_dim is its own field or, where the config omits it, hidden_size / num_attention_heads. At bf16 or fp16, an element is 2 bytes.

Here is the math for Llama 3.1, whose context limit is 131,072 tokens. For the 8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 0.125 MiB per token. Figures below are cache only, at bf16, in binary units (1 GiB = 2^30 bytes; the same values in GB are ~7% larger).

One full-context request on the 8B needs 16 GiB of cache, more than the ~15 GiB its bf16 weights occupy. And 32 users at 8,192 tokens each need twice that. A model that fits on a GPU can still run out of memory under load.

Long contexts stay expensive for two reasons. Resident memory caps concurrency, because every active request holds its cache for as long as it runs. And each decode step reads the whole cache, so tokens per second falls as context grows. Together AI's post on serving million-token context with DeepSeek-V4 walks through both at 1M-token scale.

GQA, MQA and MLA shrink the cache in the architecture

Each attention variant attacks a different term of the formula. Four designs matter:

• MHA: Multi-head attention gives every query head its own K and V head. It is the baseline and the most expensive option, the 320 GiB row above.
• MQA: Multi-query attention (Shazeer, 2019) shares one K/V head across all query heads, which divides the cache by the head count. The GQA authors later reported that it can degrade quality.
• GQA: Grouped-query attention (Ainslie et al., 2023) lets groups of query heads share a K/V head. Llama 3.1 70B has 64 query heads and 8 KV heads, which is the 8× gap between the 40 GiB and 320 GiB rows.
• MLA: Multi-head latent attention, introduced with DeepSeek-V2 (DeepSeek-AI, 2024), caches one compressed latent vector (plus a small positional key) per token per layer and rebuilds K and V from it at attention time. That shrinks the num_kv_heads × head_dim product itself. DeepSeek-V3 stores 576 elements per token per layer, against 2 × 8 × 128 = 2,048 for Llama 3.1 70B.

Sliding-window and sparse attention go after the remaining term, the token axis, by limiting how many tokens a layer keeps or reads.

PagedAttention and prefix caching manage it at serving time

The architecture fixes the bytes per token; the serving engine decides how well they get used. Fragmentation is the first problem. An engine can't know how long a reply will run, so the simple approach reserves one contiguous slab per request, sized for the maximum length. Most of it sits empty. Kwon et al. (2023) profiled existing serving systems and found that only 20.4% to 38.2% of KV cache memory held actual token state. Their fix stores the cache in fixed-size blocks and maps them through a per-request block table, the way an operating system pages virtual memory. They named it PagedAttention and built vLLM on it. Blocks are allocated on demand, so waste falls to near zero and requests can share blocks. It is exact. The attention output doesn't change.

Memory left after the weights sets your concurrency ceiling. An 80 GB H100 holds roughly 80 GiB, and Llama 3.1 8B at bf16 takes about 15 GiB of it. After a few GiB for activations and engine overhead, roughly 60 GiB is left. That is about 60 requests at 8,192 tokens, or 3 at full context.

Block sharing enables the second technique. When a request begins with tokens the engine has already processed (a system prompt, chat history, retrieved documents), it reuses their KV blocks. This is prefix caching. Matched blocks skip prefill, which cuts time to first token and cost without changing the output. A few rules decide whether you get hits:

• Matching is exact and starts at token one: The earliest change to a prompt invalidates everything after it, so a timestamp at the top kills every hit.
• Stable content goes first: System prompt, tools and documents at the top; the user's turn at the bottom.
• Hits are not guaranteed: Cached blocks get evicted under memory pressure, and a cold request pays for full prefill.

Together's research on cache-aware prefill–decode disaggregation extends the idea to a hierarchy of GPU memory, host DRAM and a cluster-wide cache over RDMA, so a prefix computed on one node can be reused on another. Together's docs list cached input pricing by model and say prompt caching is enabled by default on dedicated endpoints.

Quantization, offloading, and eviction

The next three techniques stretch a fixed memory budget, and two of them are lossy.

• KV cache quantization: Storing K and V in fp8 or fp4 cuts bytes per element by roughly 2× or 4× against bf16. It is lossy and separate from weight quantization, so verify it by evaluating LLM quality on your own task.
• Offloading: Moving cold cache from GPU memory to host DRAM, SSD or a networked store trades transfer latency for capacity. It is exact. See offloading-based inference with FlexGen.
• Eviction: StreamingLLM, H2O and SnapKV keep a subset of tokens and drop the rest. That is lossy, and methods that rank tokens by attention score clash with FlashAttention, which never writes those scores to memory.

Conclusion

Work out KV cache memory from config.json for the context length you need, then choose the GPU. Put stable prompt content first so prefix caching can hit. Benchmark tokens per second at your real context length and concurrency, not at 1K tokens.

To try open models without managing GPU memory, Together's serverless inference API is the quick start. Move to dedicated model inference when you need to choose the hardware and serving profile.

FAQ

Does ChatGPT use a KV cache? Yes, in principle. Caching K and V during decode is standard in transformer serving, and OpenAI's prompt caching docs describe storing key/value tensors, which is prefix KV reuse.

Is Redis a KV cache? No. Redis is a key-value store, a different thing that shares the name. An LLM KV cache holds attention key and value tensors in GPU memory.

Should I offload the KV cache to CPU memory? Yes, when memory is the bottleneck and the workload tolerates transfer latency: long contexts, reusable prefixes, batch jobs. Avoid it for latency-critical single-turn decode.

What is a KV cache server? A shared store, often cluster-wide, that holds computed KV blocks so other GPUs or later requests can reuse them instead of repeating prefill. LMCache is an open-source example.