DeepInfer explained: DeepSeek's open-source inference engine, and when self-hosting pays off

Updated 2026-10-20

On October 20, 2026 DeepSeek open-sourced DeepInfer, the inference engine it runs in production, under the Apache 2.0 license. The repository (github.com/deepseek-ai/DeepInfer) passed several thousand GitHub stars within days. This guide explains what DeepInfer is, what it takes to run, what it does not do yet, and how to decide between self-hosting and calling DeepSeek V4.1-Flash through an API.

Facts below are taken from the DeepInfer repository and DeepSeek's announcement as of 2026-10-20. The project is moving quickly; check the repository for the current state before you plan hardware.

What DeepInfer is

DeepInfer is the engine DeepSeek uses to serve its own API. The open-source release targets a single model: DeepSeek-V4.1-Flash (Hugging Face: deepseek-ai/DeepSeek-V4.1-Flash-1017), a Mixture-of-Experts model with about 763B total parameters, 384 experts with 6 active per token, a 1M-token context window, and MIT-licensed weights.

Its main features:

• DSpark speculative decoding — draft tokens are verified with a variable length based on confidence, instead of a fixed 5 draft tokens. DeepSeek reports 4–7% higher throughput than the fixed-length approach.
• Asynchronous scheduling to keep GPUs busy between steps.
• Structured output — constrained decoding with regex, JSON and EBNF grammars.
• Batch-invariant kernels and deterministic sampling, so the same request produces the same output regardless of what else is in the batch. Useful for evaluation and debugging.

DeepSeek also open-sourced deepseek-recipe (Rust, MIT), which converts between the Chat Completions, Responses and Messages protocols and tokens, matching the official API's behavior and guarding against special-token injection.

Hardware: what it runs on

DeepInfer supports NVIDIA SM100 GPUs (B200 / B300) and Huawei Ascend 950. It does not support H100, H200 or H20.

The smallest efficient deployment DeepSeek describes is 64 GPUs for prefill plus 64 GPUs for decode — 128 accelerators before you serve your first request efficiently.

Published throughput at 131k context:

These numbers assume high, steady concurrency. Throughput per GPU drops sharply when a cluster is underused.

What's not in the open-source release

The release is an offline batch inference engine. To run it as a production API you still need to build:

• An HTTP serving layer — there is no built-in OpenAI-compatible server.
• A KV cache system. In DeepSeek's own deployment the KV cache lives in 3FS (on the order of 100 PiB) and is kept for 3 days for agent traffic and 7 days for chatbot traffic. Prompt caching is what makes long agent sessions cheap; without it every turn re-processes the full context.
• Prefill/decode disaggregation and load balancing across nodes.

Framework support elsewhere is still early: SGLang needs a preview image for V4.1-Flash, and we found no official vLLM support statement at the time of writing.

Self-hosting vs an API: how to estimate

Self-hosting pays off only when you can keep a large cluster busy around the clock. A rough monthly estimate:

Self-hosted monthly cost (minimum efficient unit)

128 GPUs × your GPU-hour rate × 720 hours

At an illustrative $4 per GPU-hour that is about $368,640 per month before engineering time, the serving and caching layers you must build, observability, and failure recovery. DeepSeek reports its own cluster processes about 155 trillion tokens a day — the scale at which this hardware is efficient.

For most teams, the comparison is not close: unless you consistently push hundreds of billions of tokens a month and have the engineering capacity to run the missing pieces, calling V4.1-Flash through an API is cheaper and you get prompt caching, retries and an HTTP endpoint on day one.

Questions that decide it:

1. Volume — can you keep 128 B200-class GPUs busy 24/7?
2. Caching — does your workload (agents, long documents) depend on prompt caching? Self-hosting means building it yourself.
3. Data control — do you have a hard requirement that data never leaves your infrastructure? That is the strongest reason to self-host.
4. One model or many — DeepInfer serves one model. Most products also call Claude, GPT or Gemini for some tasks.

Using DeepSeek V4.1-Flash through OneFast API

OneFast API serves deepseek-v4.1-flash behind the same API key as Claude, GPT, Gemini, Kimi, GLM, Qwen and image/video models, with OpenAI-compatible and Anthropic Messages endpoints, automatic prompt caching and automatic failover between routes. Pricing is pay-as-you-go in USD — see the DeepSeek V4.1-Flash page for the live price.

cURL

curl https://api.onefast.ai/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "Summarize what DeepInfer does in two sentences."}]
  }'

As open-source releases like DeepInfer make it easier for inference providers to serve DeepSeek models, more independent routes will become available. OneFast API routes requests across multiple providers and is working toward showing measured latency and throughput per route, so you can choose on price and speed.

Summary

• DeepInfer is DeepSeek's production engine, open-sourced under Apache 2.0, serving only DeepSeek-V4.1-Flash.
• It needs B200/B300 or Ascend 950 hardware and a 64 + 64 GPU unit to run efficiently.
• The release has no HTTP server, KV cache system or load balancing — you build those.
• Self-hosting makes sense at very high, steady volume or with strict data-residency needs; for everyone else an API is the faster and cheaper path.

Sources: DeepInfer repository (README and doc/inference_deployment_qa.md), DeepSeek announcement, DeepSeek-V4.1-Flash on Hugging Face.