Inside DeepInfer: How DeepSeek Organizes Inference at Scale

Waiting for an AI response is a very ordinary experience. The first words take a while to arrive, then the answer either flows or crawls. Keeping that experience consistent when many requests arrive together is a much harder engineering problem. DeepInfer opens up part of DeepSeek’s work on that problem, giving developers code they can inspect, benchmark, and build on.

What is available today

As of October 22, 2026, the open release provides offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950 hardware. Its engine includes parallel execution, scheduling, and decoding optimizations. A complete online service still needs supporting systems, including KV cache management, disaggregated serving, and load balancing across replicas. That scope matters when estimating an integration. Official project overview.

Two stages, different pressures

Prefill processes the input context; decode generates the output that follows. A long document and a long answer put different pressures on an inference system. DeepSeek’s deployment notes describe separate vision encoding, prefill, and decode resources, with data parallelism for attention and expert parallelism for MoE computation. This makes it possible to manage the wait for the first token separately from the pace of subsequent tokens. Deployment architecture.

For a team planning its own deployment, that suggests a useful starting point: identify where users are waiting. Buying more accelerators before understanding the workload can leave the original bottleneck intact. Input length, concurrency, communication, and cache behavior all deserve a place in the experiment.

Making better use of the time between steps

Two details are especially interesting. Asynchronous scheduling overlaps CPU preparation with device execution. DSpark uses confidence estimates to allocate verification work across draft tokens. One reduces idle gaps; the other spends verification compute selectively. Their gains depend on the workload and should not be multiplied into a blanket speedup claim. DSpark and scheduling details.

This is the part of inference engineering that can disappear behind a single tokens-per-second number. A fast operation is useful only when the surrounding system can keep it supplied with work. Looking at the gaps between operations can be as revealing as looking at the operations themselves.

A few other features deserve a place in an evaluation plan. The official feature notes document their settings:

Feature	What to check
EP and EPLB	Distribute MoE experts and balance their load, while accounting for communication costs.
SWA bounded replay	Reduce decoder prefill work by replaying a limited suffix. The default 140-token setting is a retained suffix, not a context limit.
Structured output	Constrain generation with regex, JSON Schema, or EBNF.
Batch invariance and deterministic sampling	NVIDIA only. Deterministic sampling requires DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed.

Reading the performance charts without losing the context

The README reports prefill measurements on four SM100 GPUs and decode measurements on 64, each within one scale-up domain. These charts describe serving performance under specific conditions. They do not measure reasoning quality or predict results on a different machine. Official benchmark overview.

The prefill chart plots milliseconds per prefill step against non-cached input tokens per second per GPU. A request can also encounter queueing, multiple processing steps, and transport delays. Its horizontal axis therefore should not be relabeled as end-to-end time to first token. Axis definitions.

The decode chart makes the trade-off between individual speed and aggregate throughput visible. Both axes use an effective token count of 1 + 3 × 0.75 = 3.25 per step. That acceptance assumption is part of the result; a different request mix can change it. Pick an acceptable user experience first, then examine the throughput available at that operating point. Conversion logic.

A result is more useful when it can be questioned

Even the minimal offline example needs Linux, GCC 13 or newer, a clean and activated Python 3.12 environment, and the supported hardware stack: CUDA 13 for NVIDIA SM100, or CANN 9.2 for Ascend 950. These commands select NVIDIA; use ascend for Ascend. Reserve disk space for the default checkpoint download, or set DEEPINFER_HF_MODEL_DIR to a local copy. Official prerequisites.

git clone --recursive https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

The repository includes a sweep script that varies batch settings, records results, and produces curves. Its example documentation explains warmup, steady-state measurements, and profiler interference. For a useful reproduction, keep the commit, hardware layout, context length, and full configuration alongside the output. Those records make a surprising result much easier to investigate. Sweep script · Measurement guidance.

Quality needs its own check. The eval/ directory provides offline workflows for AIME 2026, HLE, and GPQA Diamond, separating generation from scoring. After tuning the engine, rerun a consistent quality evaluation instead of treating a higher throughput figure as sufficient evidence of improvement. Evaluation workflow.

Who should take a closer look

Infrastructure teams with the supported hardware can start with offline examples and a workload shaped like their real traffic. Teams building API integrations should also inspect deepseek-recipe, which handles conversions between requests, model prompts, and responses. The caller supplies inference, tool execution, and HTTP transport. DeepSeek Recipe.

For application developers, the useful takeaway is a better set of questions: how long does a long input wait, how quickly does the answer arrive afterward, and what happens under concurrency? DeepInfer gives those discussions code and measurement methods to refer to. This article reviews the official material available on October 22, 2026; it does not report an independent benchmark.