Skip to content
audiotale.net
Sign in
NewsNews

Cloudflare GLM-5.3 Workers AI leaked 155 responses via an SGLang KV cache bug

Cloudflare says long prompts to GLM-5.3 received other customers' data between 27 August and 9 September; the fix is SGLang v0.5.22 and a KV pool that fits the context window.

audiotale.net team

· 6 min read

Written by AI from the sources linked below. How we use AI

On this page
  1. Key points
  2. What to do
  3. What happened
  4. How the leak worked
  5. The numbers
  6. Background
  7. What it means
  8. What we don't know yet
  9. What to watch next
  10. Sources

Cloudflare has published an incident report saying that between 27 August and 9 September 2026, 155 responses from the GLM-5.3 model on its Workers AI platform contained data from other customers' earlier requests. The cause was a mismatch between the model's advertised context window and the memory set aside for its attention cache, combined with a bug in the open-source inference engine SGLang. Cloudflare says it found no evidence of deliberate exploitation. Anyone running SGLang in disaggregated prefill/decode mode should check their configuration and upgrade to v0.5.22 or later.

Key points

  • 155 responses were affected, all to prompts longer than 681,280 tokens, on the @cf/zai-org/glm-5.3 model, from its launch on 27 August until the model was restored on 9 September (Cloudflare).
  • Cloudflare configured a 1,048,576-token context window but its prefill GPUs' KV cache pool held only 681,280 tokens.
  • An SGLang bug turned the mismatch into a leak: over-length requests were swapped for a one-token placeholder, but still handed to the decode node, which read stale cache pages left by earlier requests.
  • Security researcher Gadi Evron of Knostic reported it on 8 September; Cloudflare took the model offline within 40 minutes of declaring an incident and restored it the same day.
  • Cloudflare recommends all SGLang operators update to v0.5.22 or later. It says no other Workers AI model was affected.

What to do

If you use GLM-5.3 on Workers AI: Cloudflare says a small number of customers received responses containing other customers' data, and only for prompts above 681,280 tokens. If you sent prompts that long to that model between 27 August and 9 September, consider them potentially exposed: review what they contained, and treat responses from that period with long prompts as possibly containing someone else's text. The report does not say how affected customers are being notified, so check your account contacts and Cloudflare's incident communications.

If you run SGLang in disaggregated mode (separate prefill and decode GPU pools), Cloudflare recommends:

  1. Compare the KV pool size printed in the engine's startup log with your context length on every prefill and decode node, after any change to weights, quantization, parallelism or memory fraction. If the pool is smaller, lower the context length to fit.
  2. Upgrade to v0.5.22 or later, which contains the admission fixes (#36988 and #38935) so over-length requests are rejected before handoff.
  3. Keep deferred KV release enabled on both roles (default after #41023) and bound Mooncake transfer batches.
  4. Turn on --disaggregation-enable-kv-checksum and --kv-canary where your model supports them; Cloudflare notes they do not yet cover every architecture.
  5. Make sure your load balancer lets requests proceed to the first token so clients disconnecting do not trigger cross-node aborts.

How to check whether you are exposed: the bug needs three things at once: disaggregated prefill/decode, a KV pool smaller than the context length, and a prompt longer than the pool. A single-node deployment returned an error instead, Cloudflare says.

What happened

Cloudflare launched GLM-5.3, a text-generation model from Z.ai built for agentic coding, on Workers AI on 27 August 2026. It serves the model with SGLang in a disaggregated configuration: prefill GPUs process the prompt and build its KV cache (the stored attention keys and values for the prompt), then pass that cache over RDMA, via Mooncake, to decode GPUs that generate the answer.

On 8 September at 23:11 UTC a report arrived through Cloudflare's HackerOne programme: very long prompts sometimes returned text belonging to other users' sessions. Cloudflare began verifying at 08:00 UTC on 9 September, declared an incident at 13:22, reproduced the leak with two independent accounts within 25 minutes, and set the model to private at 14:00 so requests returned HTTP 400. All prefill nodes were moved to a corrected configuration by 14:52, and the full deployment was rebuilt with an engine patch by 19:04. The model returned to public service at 19:40.

Cloudflare says that after excluding its own testing and the researcher's, the requests it identified were consistent in timing and volume with ordinary long-context use rather than someone repeatedly exploiting the bug. Cloudflare and SGLang published coordinated disclosures on 26 October.

How the leak worked

The KV cache lives in GPU memory as a pool of fixed-size pages. Pages are not zeroed between requests; the next request simply overwrites what it needs. That is normal for inference engines and important for speed, but it makes the engine responsible for knowing which pages a request has written.

Cloudflare's account of the failure, step by step: the router told the decode node the prompt length (eight pages in its diagram) and decode reserved eight pages. The prompt was too long for the prefill pool, so SGLang's scheduler "rejected" it by recording an error and swapping the prompt for a one-token placeholder. In single-node mode the placeholder drained through and the client got an error. In disaggregated mode both admission queues checked whether a request was finished, not whether it was rejected, so the rejected request looked live. Prefill ran the placeholder, sent one page of KV to decode and signalled that the transfer was complete. Decode read all eight reserved pages, but only one had been written by this request. The other seven still held earlier users' prompts and partial generations. The client received HTTP 200, a coherent answer and a prompt_tokens count matching the long prompt.

Cloudflare adds that the bug had been reported to SGLang by another user as issue #30233 on 6 July 2026, which it noticed during the investigation.

The numbers

GLM-5.3 prefill KV pool versus context length (tokens)tokens
GLM-5.3 prefill KV pool versus context length (tokens) (tokens)
tokens
Fixed prefill KV pool (about)2040000
Context window1048576
Original prefill KV pool681280

Source: Cloudflare incident report

The original pool covered about 65 percent of the context window (681,280 of 1,048,576 tokens), so prompts between those two sizes could trigger the rejection path. Cloudflare's fix copied its GLM-5.2 configuration, which freed GPU memory so the pool holds about 2.04 million tokens, roughly twice the context length. Cloudflare says it was aware of the size mismatch before launch, but its pre-launch tests checked HTTP status codes and performance rather than validating output coherence, so they did not catch it.

Background

KV-cache memory-safety bugs are a known class: Cloudflare says it has been flagging such issues to SGLang since v0.5.10, including earlier page-lifetime and cross-node race problems around offload and aborted transfers. In each case a page is freed, reused or read while something else still has a claim on it, and the result is corrupted output or one user's data in another's response with no error raised.

What it means

For AI platform operators, the report is a checklist: validate output, not just status codes, in long-context tests; treat the KV pool versus context length as a deploy-time invariant; and do not rely on an inference engine to reject what does not fit. For customers of hosted inference, it is a reminder that tenant isolation can fail below the model, in scheduling and memory management, and that very long prompts are the least-exercised path. For SGLang users, the fix is available now; the longer-term work is for the engine to detect at read time that a request is attending over a page it does not own. Cloudflare describes a prototype that tags every physical page with a value changed on reallocation.

What we don't know yet

  • Which customers received other customers' data, what that data was, and how affected customers were told. The report gives counts, not details.
  • Whether non-Cloudflare SGLang deployments were exposed to the same bug. Cloudflare says the issue needs a pool smaller than the context length, but it cannot know others' configurations.
  • Why the exposure ran for about two weeks before a report: Cloudflare's tests did not catch it, and the report credits the outside researcher.

What to watch next

Watch for SGLang's own disclosure and release notes, further KV-checksum and canary coverage for GLM 5.3, GLM 5.3 Flash and DeepSeek V4 (which Cloudflare says it is working on), and whether other hosted-inference providers publish similar checks.

Sources

SaveStart the discussion

Written by

audiotale.net team

The official account of the people who run audiotale.net: announcements, conversation starters and the newsroom's drafts. Posts written with AI say so. Every other account here is a real person.

Read next