Cerebras explains how Kimi K3 fits on 59 wafers: 93 layers, 44 GB each

Cerebras has published a step-by-step account of how it fits a model of roughly three trillion parameters onto its wafer-scale chips, using Moonshot AI's open Kimi K3 as the worked example. The short version: it splits the model's 93 layers across 59 wafers, keeps every weight in on-chip memory so nothing is fetched from outside, and passes only small activations between wafers. The post is a rare look at how a non-GPU inference provider sizes a frontier model, and the arithmetic is simple enough to check yourself.

Key points

• Cerebras says a single WSE-3T wafer holds 44 GB of on-chip memory, while trillion-parameter models need terabytes for weights alone.
• For Kimi K3, which it describes as having 69 linear-attention layers and 24 dense layers, Cerebras's worked example comes to 59 wafers: 35 for the linear layers and 24 for the dense ones.
• Only activations cross between wafers, using RDMA over Ethernet at roughly 2 microseconds per hop, according to Cerebras.
• Cerebras says this layout serves multi-trillion-parameter models at 1,000 to 2,000+ tokens per second; that is the company's claim, and we could not verify it independently.
• The post is an illustrative example, not a description of Cerebras's production layout, which it says uses finer-grained stages.

What happened

On 22 October 2026 Cerebras engineer Sarah Chieng published "How we deploy Trillion-parameter frontier models on Cerebras" on the company's blog. It answers a question the company says it gets often: how can a chip with a fixed amount of on-chip memory run a model whose weights are many times larger?

The answer is to stop thinking about one chip. Cerebras says a single wafer holds 44 GB of on-chip memory, and the model is spread over many wafers, each acting as one or more stages of a pipeline. The weights for each stage stay where they are; the request's intermediate results move from wafer to wafer.

The post makes its assumptions explicit, which is what makes it useful. It uses Kimi K3 as the example, with a 256k-token average request, 4-bit weight quantisation and an 8-bit KV cache (the stored keys and values attention needs for tokens already processed). It refers to Hugging Face's implementation as the example of 4-bit weights.

Background

Large language models are normally served on GPUs, where weights sit in high-bandwidth memory next to the processor. Cerebras takes a different route: its WSE-3 chip, a full silicon wafer, carries 44 GB of SRAM on the die itself. The company has said in earlier materials that the chip has four trillion transistors and 900,000 cores; those figures are widely repeated in trade coverage. The trade-off is capacity. 44 GB is far less than a frontier model needs, so models have to be partitioned across many wafers.

Kimi K3 is a good test case because it is large and open. Moonshot AI's own launch post describes it as a 2.8-trillion-parameter model with native vision, a 1-million-token context window and a mixture-of-experts design that activates 16 of 896 experts. Moonshot calls it the first open model in the "3T class" and says it still trails the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol. The Hugging Face repository moonshotai/Kimi-K3 reports a total of about 2.78 trillion parameters in its safetensors metadata, which fits the 2.8 trillion figure. Cerebras rounds this to "about 3 trillion" in its post, a small difference worth knowing about when you redo the sums.

We covered Moonshot's later release in our article on Kimi K3.1 Pro.

How it works

Cerebras walks through six steps. We repeat its arithmetic here and show where our check agrees or differs.

1. Size each layer. At 4 bits (half a byte) per parameter, 3 trillion parameters occupy about 1.5 terabytes. Divided across 93 layers (69 linear plus 24 dense), that is roughly 16 GB a layer, which is the figure Cerebras uses. Using Moonshot's 2.8 trillion instead gives about 1.4 terabytes and about 15 GB a layer. The conclusion below does not change.

2. Count the wafers. The two layer types differ in what else they need on the wafer:

• A dense (full-attention) layer stores keys and values for every token of every active request, so it needs a large KV cache. Cerebras places one dense layer per wafer and leaves the rest of the 44 GB for that cache.
• A linear-attention layer keeps only a small fixed-size state per request, whatever the context length, so two layers fit on one wafer (2 x 16 GB = 32 GB, under 44 GB).

That gives 24 wafers for the dense layers and 35 for the 69 linear layers (69 divided by 2, rounded up), a total of 59. Our own check with the smaller 15 GB layer size still yields 59: a third linear layer would need about 45 GB, more than a wafer holds.

3. Connect the wafers. Cerebras runs the model as a pipeline. Because each layer (about 16 GB) fits inside a wafer's 44 GB, whole layers sit on one chip and weights never move. Only activations travel, over RDMA on Ethernet, at roughly 2 microseconds per hop. With 59 stages that is on the order of 120 microseconds of pure link time per token for one trip through the pipeline, by our multiplication of the company's figure; it ignores compute and any extra hops, so treat it as a lower bound rather than a measurement.

4. Serve one request. Wafer 1 runs the first layers and hands its activations to wafer 2, and so on until the last wafer produces the next token.

5. Serve many requests. A pipeline running one request at a time would leave most wafers idle. Cerebras instead keeps the pipeline full: while stage 3 handles request A, stage 2 handles request B and stage 1 handles request C. Each stage can also process a micro-batch, a small group of requests. Smaller micro-batches make each request faster; larger ones raise total throughput. In production, the company says, stages are much finer-grained than one per layer.

6. Count concurrent requests. The memory left on each dense-layer wafer after its 16 GB layer, roughly 28 GB, is what holds KV cache, and it caps how many 256k-token requests can be in flight. The post gives the method but the final figures appear in graphics we could not read as text, so we do not quote a concurrency number.

Total59

Linear-attention layers (2 per wafer)35

Dense layers (1 per wafer)24

Wafers needed for Kimi K3 in Cerebras's worked example (wafers)

	wafers
Total	59
Linear-attention layers (2 per wafer)	35
Dense layers (1 per wafer)	24

Source: Cerebras blog, 22 October 2026

Linear-attention69

Dense24

Layers in Kimi K3 by type (layers)

	layers
Linear-attention	69
Dense	24

Source: Cerebras blog, 22 October 2026

What it means

For developers choosing an inference provider. The post makes one point clear: speed on this hardware comes from memory placement, not from clever kernels. If the entire model sits in on-chip SRAM, token generation is not limited by fetching weights, which is the usual bottleneck on GPUs. Cerebras says that is how it reaches 1,000 to 2,000+ tokens per second on multi-trillion-parameter models. Those are the company's numbers.

For model designers. The layout rewards linear-attention layers. They need no per-token cache, so two fit on a wafer, while each dense layer consumes a whole wafer. A model with fewer dense layers needs fewer chips for the same parameter count. Kimi K3's mix of 69 linear and 24 dense layers is, by this logic, cheaper to host on Cerebras than a model that is all dense attention. Moonshot says the model uses Kimi Delta Attention; Cerebras does not claim that this was a design goal.

For the market. Cerebras closes its post by arguing that the useful measure is throughput per unit of power, such as tokens per second per megawatt at the same generation speed, not concurrency or batch size alone. It gives no such figure in this post. Readers comparing Cerebras with GPU clusters should ask for it, and for independent tests.

What we don't know yet

• Real concurrency and cost. The post's concurrency calculation and any power or price figures are not given as text. We do not know how many users 59 wafers serve, or what that costs.
• Whether this is the production layout. Cerebras calls this "an illustrative example" and says production stages are finer-grained.
• Independent speed tests. The 1,000 to 2,000+ tokens per second claim is from Cerebras. Its page also carries a notice that speed comparisons are based on third-party benchmarking or internal testing and may vary by workload, configuration and date.
• The model's size. Cerebras says about 3 trillion parameters; Moonshot and the Hugging Face metadata say about 2.8 trillion.
• Benchmark scores. The post quotes Kimi K3 benchmark results against GPT-5.6 Sol. Those are Moonshot's evaluation numbers, repeated by Cerebras; we have not verified them.

What to watch next

Look for independent measurements of tokens per second and tokens per megawatt on Cerebras's service for models of this size, and for whether other open models adopt more linear-attention layers. Also watch whether Cerebras publishes the concurrency and power figures that this post stops short of.

Sources

• Cerebras: How we deploy Trillion-parameter frontier models on Cerebras (primary source)
• Moonshot AI: Kimi K3, Open Frontier Intelligence
• Hugging Face: moonshotai/Kimi-K3
• SDxCentral: Cerebras unveils four-trillion-transistor chip