A new open-source project called DeepInfer has been released to support high-throughput inference for large language models in production environments. Developed by DeepSeek-AI, the framework is designed to handle the computational demands of serving models at scale, particularly those based on the DeepSeek architecture.

The software provides offline inference capabilities and includes core components necessary for large-scale serving, such as optimized kernel libraries, parallel execution mechanisms, and deterministic sampling controls. It supports deployment on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with requirements for additional infrastructure including a KV cache system and disaggregated prefill/decode components.

Key technical features include FlashMLA and DeepGEMM-accelerated matrix operations, TileKernels for memory-efficient processing, and asynchronous scheduling that overlaps CPU batch management with GPU execution. The framework also implements batch-invariant inference to ensure consistent outputs regardless of input batch composition, and offers structured output generation through regex, JSON schema, or EBNF grammar constraints.

Deployment configuration involves environment variables for performance tuning, such as DEEPINFER_ENABLE_MTP_VARLEN for variable-length token handling and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS for controlling retained context windows during speculative decoding. Benchmark results indicate competitive performance across multi-GPU setups, with measured throughput reported for both prefill and decode phases using 4 SM100 GPUs for prefill and 64 for decode operations.

The project is distributed under the Apache License 2.0 and builds upon prior work from vLLM, SGLang, and TensorRT-LLM communities. Documentation includes setup instructions requiring Linux, GCC 13.0, Python 3.12, and either CUDA 13 or CANN 9.2 depending on hardware. Example implementations demonstrate offline inference workflows and evaluation methodologies.

DeepInfer represents a technical advancement in reducing inference latency and improving resource utilization for large model serving, particularly in environments where deterministic behavior and structured output generation are critical requirements.