News › ai-infrastructure
By The Linux Camp News, October 23, 2026
DeepInfer, DeepSeek's inference framework, is released under Apache 2.0.
DeepSeek has made DeepInfer available as an open-source offline inference framework supporting DeepSeek-V4.1-Flash, according to its repository retrieved October 23, 2026. DeepSeek says the framework powers its large-scale production services, but describes the public release as offline inference with core scaling features, not the complete production service. The supplied README does not establish an exact release date or release tag. 1
Hardware compatibility is the first evaluation gate. The documented targets are NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The prerequisites specify Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 for NVIDIA or CANN 9.2 for Ascend. Those requirements describe a narrow deployment target; they do not establish support for other accelerators. 1
Check dependency revisions before settling on a CUDA image. In the October 23 source snapshots, DeepInfer lists CUDA 13, while FlashMLA lists CUDA 13.1 or newer and recommends 13.2 or newer. FlashMLA explains that 13.1 uses a slower conversion fallback. Without DeepInfer's pinned dependency revisions, the packet cannot resolve the effective minimum toolchain; the top-level prerequisite alone is not enough to settle that choice. 1 2
DeepInfer's DSpark path accelerates decoding, the output-generation stage, using draft tokens: provisional tokens that undergo verification. Confidence estimates determine verification scheduling. The documented switch is DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, defaulting to five. The README describes the mechanism but does not explain how confidence is calculated or quantify the speedup. 1
Bounded replay reduces a different source of work: decoder prefill replays only a suffix, the final portion of the token sequence. For the default V4.1 settings, the README specifies DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. That retained length is constrained, not an arbitrary tuning knob: it must cover the SWA window plus maximum backtracking length. Reducing it without checking those requirements would depart from the documented configuration rule. 1
For repeatable tests, distinguish batch invariance from deterministic sampling. DeepSeek documents DEEPINFER_BATCH_INVARIANT=1 as keeping results identical across batch compositions. Identical sampled outputs require that setting, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed passed to SamplingParams. Thus, the documented deterministic configuration disables the variable-length draft-token path rather than merely adding a seed. Both guarantees are NVIDIA-only. 1
For service operators, the important boundary is what still needs integration. DeepSeek explicitly excludes a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for prefill and decode. It says most engine functionality needed for disaggregation is implemented, but still lists the service capability as missing. The operational inference is that this release offers an engine to evaluate, not a complete replacement for an inference-serving stack. 1
Performance evidence also has a boundary. DeepSeek documents measurements on four SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. The supplied excerpt contains no numerical results or comparison baseline. Those hardware counts describe the benchmark setup, not established minimum deployment requirements, and the excerpt cannot establish an end-to-end advantage or equivalent Ascend performance. 1