What is a KV cache? How LLM servers store attention and why it can leak

In short

A KV cache is the memory an LLM server keeps for each active request: for every token already read or written, it stores the "key" and "value" vectors that attention needs, so the model does not recompute them for each new token. It is what makes text generation fast, and it is also large and shared hardware state, so serving engines manage it as pages. When a page is reused while something still points to it, one user's data can show up in another's answer, as happened in the Cloudflare GLM-5.3 incident.

Why a cache exists at all

A transformer language model produces text one token at a time. To choose the next token, each attention layer compares the newest token's "query" against the "keys" of every earlier token, then mixes the matching "values". The keys and values of earlier tokens do not change when a new token is added, because the model only looks backwards. Recomputing them for the whole conversation at every step would waste almost all the work, so the server computes them once, keeps them in GPU memory, and appends the new token's keys and values each step. That stored set is the KV cache.

The trade is compute for memory. The cache grows with the number of tokens, the number of layers and the number of attention heads, and every active request has its own. The vLLM paper puts the problem plainly: the KV cache memory for each request "is huge and grows and shrinks dynamically", and managing it inefficiently wastes memory through fragmentation and duplication, which limits how many requests fit in a batch (Kwon et al., 2023).

Two phases: prefill and decode

Serving a request has two distinct stages:

• Prefill reads the whole prompt in parallel and builds the KV cache for it. It is compute-heavy, and its duration sets the time to the first token.
• Decode generates the answer one token at a time, reading the whole cache at each step and appending to it. It is limited more by memory than by arithmetic, and its speed sets the time per output token.

Because the two stages stress hardware differently, some operators run them on separate GPU pools. The DistServe paper describes this "disaggregation": it assigns prefill and decoding to different GPUs to remove interference between them, and reports serving 7.4 times more requests under latency targets in its tests (Zhong et al., 2024). The price is that the cache built on a prefill GPU must be shipped to a decode GPU, typically over a fast network link (Cloudflare's setup uses RDMA through a library called Mooncake). That handoff is a new place for things to go wrong.

Paging: how engines manage the cache

Early systems reserved one contiguous block of memory per request, sized for the longest possible output, and much of it sat unused. PagedAttention, the idea behind vLLM, borrows from operating systems: split the cache into fixed-size pages, hand them out as a request grows, and let the attention computation read pages that are not next to each other. The paper reports near-zero waste in KV cache memory and 2–4 times higher throughput than earlier systems at the same latency (Kwon et al., 2023).

SGLang, the engine in the Cloudflare incident, uses a related design and adds RadixAttention, which reuses cached prefixes across requests (for example a shared system prompt) so common text is not recomputed. Its authors report up to 6.4 times higher throughput than other systems on tasks such as agent control, JSON decoding and multi-turn chat (Zheng et al., 2023).

Paging has a consequence worth stating: a pool of pages is shared by every request on that GPU. Cloudflare's report notes that the pages "are not currently zeroed between requests": the next request that is given a page simply overwrites it. That is a deliberate speed choice, common across inference engines, and it puts the burden on the engine to track which pages each request has actually written.

Sizing: context length versus pool size

Two numbers must agree. The context length is the longest prompt-plus-answer the service promises to handle. The KV pool is how many tokens' worth of cache fit in GPU memory after the model weights are loaded. SGLang sets the pool size at start-up from the memory left over, so nothing forces it to match the advertised context length.

Context window1048576

Original prefill KV pool681280

Cloudflare GLM-5.3 prefill pool versus context window (tokens) (tokens)

	tokens
Context window	1048576
Original prefill KV pool	681280

Source: Cloudflare incident report

In the Cloudflare case the context window was 1,048,576 tokens but the prefill pool held 681,280. Prompts between the two sizes could not fit, and the engine had a code path to reject them. That path, in disaggregated mode, had a bug.

What went wrong in the incident

Reduced to the mechanism, per Cloudflare's report (source):

1. The decode node reserved pages for the full length of the prompt.
2. Prefill rejected the oversize prompt but, instead of aborting, substituted a one-token placeholder and sent that one page of cache across, marking the transfer complete.
3. The decode node assumed the full prompt's cache had overwritten its pages. It had not; only one page was fresh.
4. Decode read all the reserved pages. The rest held earlier users' prompts and partial answers, and the model attended over them.

The result was an ordinary-looking HTTP 200 response with a coherent answer containing content from other sessions. Cloudflare counted 155 affected responses between 27 August and 9 September 2026, all to prompts over 681,280 tokens.

How to avoid this class of bug

If you operate inference, Cloudflare's own list of recommendations is a good checklist: compare the KV pool in the start-up log with your context length after any change to weights, quantization, parallelism or memory settings; upgrade SGLang to v0.5.22 or later; keep deferred KV release on; enable the engine's KV checksum and canary options where your model supports them; and avoid load-balancer aborts mid-transfer. The more general lessons:

• Test output, not just status codes. Cloudflare says its long-context tests looked only at HTTP codes and performance, so a response that was fast and wrong passed.
• Treat memory reuse as a security boundary. Unzeroed, reused pages are fine only if every read is guaranteed to follow a write by the same request. Cloudflare describes work toward tagging pages so a request can detect it is reading a page it does not own.
• Make limits fail closed. A request that cannot be served should be rejected before any memory is reserved or transferred, which is what the SGLang changes now do.

If you only use hosted models, the practical takeaway is narrower: confidential data in very long prompts depends on the provider's isolation below the model, and incident reports like this one are how you learn how that isolation is built.

Common questions

Does the KV cache store my conversation? It stores numerical keys and values derived from your tokens, for as long as the request or session needs them. Engines may also keep prefixes for reuse, which is a provider policy question; read the provider's data-handling terms.

Is the cache the same as the model's memory? No. It is temporary, per-request working state. The model's weights do not change while serving.

Why do long contexts cost more? The cache grows with every token, so long prompts take more GPU memory and leave room for fewer simultaneous requests.

Can I see the cache size? On self-hosted engines, usually yes: Cloudflare notes SGLang prints the selected KV pool size at start-up and exposes it as a metric.

Sources

• Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs
• Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving