DeepSeek open-sources DeepInfer, the engine behind its production serving

DeepSeek has open-sourced DeepInfer on GitHub, describing it as a high-performance, large-scale inference framework for its own models and, more pointedly, as the engine that powers its large-scale production services. The release covers the offline inference path together with the core capabilities needed to serve at scale.

The form of the release matters as much as the contents. This is not a model card or a technical report but buildable code: an Apache 2.0 repository with build instructions and configuration notes, so developers can compile and run the framework themselves.

The feature list targets the unglamorous parts of serving. DeepInfer ships optimized kernels plus a model and sampler runtime, attention-side data parallelism with balanced context parallelism, MoE expert parallelism with EPLB load balancing, and asynchronous scheduling that overlaps CPU-side batch scheduling with device execution.

Several capabilities aim at reproducibility rather than raw speed. Batch-invariant inference keeps results identical across different batch compositions; deterministic sampling returns the same output for the same input tokens and sampling parameters; and structured outputs constrain generation to a regex, JSON schema or EBNF grammar. In production those properties often decide whether a system is usable at all.

Hardware support is the second signal. The project lists both Nvidia GPUs and Huawei Ascend NPUs, which suggests the engine was designed with domestic accelerators in mind rather than retrofitted to them later.

The boundaries are explicit. DeepSeek says the release does not include a KV cache system, prefill and decode disaggregation, or load balancing across data-parallel replicas. Most engine functionality is implemented, but anyone assembling a full service still has to supply the cache and routing layers.

Why it matters: open-sourcing the serving engine moves the competition beyond model weights and into cost per token. Whoever serves the same throughput on fewer chips sets the price floor, and outside teams can now reproduce DeepSeek's throughput and latency claims instead of taking them on faith.

Two things to watch: whether the community fills the KV cache and prefill-decode gaps, and whether anyone ports the framework to non-DeepSeek models. If both happen, DeepInfer stops being a disclosure and starts being independent infrastructure.