DeepSeek open-sources DeepInfer, the inference framework behind its production service, minus the KV cache system

DeepSeek has open-sourced DeepInfer, the inference framework it says powers its large-scale production services, under the Apache 2.0 licence. The release appeared on 20 October 2026 and quickly drew attention on Hacker News (875 points at the time of writing). It is a real piece of production infrastructure, but it is deliberately incomplete: it runs one model, on two kinds of hardware, for offline inference, and leaves out the parts of a serving stack that arguably matter most at scale.

Key points

• DeepInfer is released under the Apache 2.0 licence and, per its README, is "the high-performance inference framework that powers DeepSeek's large-scale production services".
• This release supports one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and offers offline inference.
• It does not include a KV cache system, prefill/decode disaggregation, or load balancing across data-parallel replicas, which the README lists as required for a complete inference service.
• Features include expert parallelism with load balancing, speculative decoding with confidence-scheduled verification, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs.
• The README says DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.

What happened

DeepSeek published the DeepInfer repository on GitHub and announced it on X as "our in-house inference framework". We read the repository's README directly; the points below come from it unless we say otherwise. We could not open Bloomberg's report of the release (it blocked automated access), so everything here rests on DeepSeek's own description and on what the community has posted around it.

The README describes DeepInfer as providing "offline inference with the core features needed for serving at scale". The only supported model is DeepSeek-V4.1-Flash, and the default checkpoint is DeepSeek-V4.1-Flash-1017, which the Hugging Face page shows was created on 17 October 2026. We covered that model's release and MIT licence in our earlier article. The supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Building requires Linux, GCC 13 or newer and Python 3.12, plus CUDA 13 for NVIDIA or CANN 9.2 for Ascend, and the build is controlled by an environment variable, DEEPINFER_ACCELERATOR, set to nvidia or ascend.

The feature list, in plain terms

The README lists these capabilities. Our explanations of the general techniques are background, not claims about DeepSeek's implementation.

• Optimised kernels. The model and sampler runtime use kernels from DeepSeek's earlier open-source libraries (FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect) plus custom TileLang kernels. So part of what is "new" here is the glue that puts those previously released pieces together.
• Parallelism. The README names attention data parallelism with balanced context parallelism, and, for the mixture-of-experts layers, expert parallelism (EP) with EPLB, a load balancer for experts. In a mixture-of-experts model, each token is routed to a few experts, and when experts live on different GPUs, uneven routing leaves some GPUs idle. Balancing that is a central problem for serving large MoE models.
• DSpark and confidence-scheduled verification. The README says these accelerate decoding using draft tokens and use confidence estimates to schedule verification. Speculative decoding in general has a cheap process propose several tokens that the full model then checks in one pass. It is switched on with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length (default 5).
• Decoder SWA bounded replay. This reduces decoder prefill work by replaying only a bounded suffix of the context. The README gives default settings for V4.1 (a compact truncation mode and 140 retained tokens) and notes that the retained suffix must cover the sliding-window attention window plus the maximum backtracking length.
• Asynchronous scheduling. CPU batch scheduling overlaps with device execution; the README calls it a "zero-overhead batch scheduler" and says it is on by default.
• Batch-invariant inference and deterministic sampling. With DEEPINFER_BATCH_INVARIANT=1, results stay identical regardless of which other requests share a batch. Adding a fixed seq_seed and turning off the variable-length draft option gives identical outputs for identical input tokens and sampling parameters. Both are NVIDIA only. These matter for reproducible evaluations and for debugging.
• Structured outputs. Generation can be constrained to a regular expression, a JSON schema or an EBNF grammar, passed through SamplingParams.

What is missing

The README is unusually direct about what the release leaves out. It lists, as required for a complete inference service but not included: a KV cache system; prefill/decode disaggregation (it says most of the engine functionality needed is already implemented); and load balancing across data-parallel replicas for both prefill and decode. Hacker News commenters noticed the same gap, one noting the KV cache system is "notably missing". The practical meaning is that you can run the model with DeepSeek's engine and read how its kernels and scheduling work, but you cannot assemble DeepSeek's production service from this repository alone. That is the repository's own statement, not a commentator's.

The numbers

The README's benchmark section shows two charts as images, throughput against latency for prefill and output throughput against per-user speed for decode, measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, with a script (run_sweep.sh) to reproduce them. We could not read exact values from the charts, so we do not quote any. All benchmark figures are DeepSeek's own and should be treated as the company's claims until others reproduce them.

One more figure circulated in the Hacker News thread: a commenter quoted a post on Zhihu saying a single cluster serves 155 trillion tokens per day and earns about $38 per card per hour at off-peak pricing. We could not verify that post or what hardware it refers to, so we treat it as unconfirmed. Another commenter noted that typical H100 rental prices are far lower, but that comparison depends on the unspecified card and on pricing, so it proves nothing by itself.

Background

DeepSeek has a history of open-sourcing parts of its stack as separate libraries. FlashMLA, DeepGEMM and DeepEP, all named in the README, were released earlier as independent projects. DeepInfer is the first time, as far as the README shows, that these pieces come with a full model runtime and scheduler. That it started from vLLM is notable: the README thanks the vLLM, SGLang and TensorRT-LLM developers, which suggests the project may be easier for people who know those engines to follow.

What it means

For people running DeepSeek models. Today most providers serve DeepSeek models with general-purpose engines such as vLLM or SGLang. DeepInfer gives them a reference for DeepSeek's own configuration, including the V4.1-specific tuning values in the README. Whether it outperforms those engines in a provider's setup depends on the missing pieces, which each provider would have to supply.

For learners. The repository is a readable example of how a frontier lab combines expert parallelism, speculative decoding and scheduling tricks. Even if you never run it, the feature list is a good map of what high-throughput MoE serving involves.

For hardware. Support for NVIDIA SM100 and Huawei Ascend 950 in the same codebase is a signal about where DeepSeek expects to run models. It is not a performance comparison; the README gives none.

For policy. The release lands the same week the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products; see our report on that order. An Apache 2.0 inference engine is software that agencies and vendors will now have to ask about; the order's reach to software "developed using" DeepSeek implementations is not defined.

What we don't know yet

• Whether independent tests will reproduce DeepSeek's benchmark curves.
• When, or whether, DeepSeek will release the KV cache system and disaggregation components.
• Whether support will extend beyond V4.1-Flash to other models.
• Whether the throughput and revenue figures circulating online are accurate.

What to watch next

Watch for reproduction attempts and comparisons against vLLM and SGLang on the same model, for pull requests that upstream DeepInfer's techniques into those projects, and for the README's own list of missing components to shrink.

Sources

• DeepInfer README, DeepSeek (GitHub)
• DeepSeek-V4.1-Flash-1017 model page, Hugging Face
• DeepSeek just open-sourced its inference engine, Hacker News discussion, 20 October 2026
• DeepSeek unveils open-source tool to help cut AI operating costs, Bloomberg (headline only; we could not open the article)