Repository navigation
[Roadmap] 2026 Q4: close the gap to official DeepInfer #3
Description
English · click to expand
How to read this roadmap
- Two levels. This page groups the work by module; each item links the issue that explains it, and that issue links its PR. Finished items are checked.
- Titles put the module in brackets, then the change, e.g.
[Attention] Read sparse attention keys through row indices. A PR has the same title as its issue and closes it on merge. - Labels give the kind of change and are shown after every item:
perf·feature·bugfix·refactor·docs·ci. - Where to start
- New to the project: read the Performance timeline, then [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 for the first round of optimizations.
- Following one module: read its section; items are in the order they were done.
- Studying one change: open its issue for the data, the official approach, the change, validation and measurement; the PR has the code.
Performance timeline
Real DeepSeek-V4.1-Flash prefill, 4 × B200, EP4 / DP4 / CP1, 131,072-token context (126,976 cached), 4,096 new tokens per rank per step.
| Change | Issue | Throughput (tok/s/GPU) | Step (ms) | Share of official |
|---|---|---|---|---|
| Initial version | — | 1,084.8 | 3,775.9 | 3.0% |
| Tensor-core sparse attention | #1 | 2,571.7 | 1,592.7 | 7.1% |
| Fused index scoring | #1 | 3,373.1 | 1,214.3 | 9.3% |
Local-row MoE combine (current main) |
#1 | 4,315.9 | 949.0 | 11.9% |
| bf16 gate GEMM (PR open) | #4 | 4,589.4 | 892.2 | 12.7% |
| Indexed attention keys (PR open) | #7 | 5,004.4 | 817.7 | 13.8% |
| Official DeepInfer | — | 36,170.9 | 113.2 | 100% |
The two open rows were measured in one job against main (4,322.1 tok/s/GPU in that job) and are not yet combined. Earlier rows come from separate jobs (#1). The official number is from an earlier job, so the shares are approximate.
Attention
DeepInfer-Lab 281 ms per step (key gathering 182 + kernel 99) · official 14.9 ms (FlashMLA reads paged KV by index)
- Run sparse attention on tensor cores — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - Read sparse attention keys through row indices — [Attention] Read sparse attention keys through row indices #7 (−130 ms measured) ·
perf - Profile the indexed kernel, then close the kernel gap to FlashMLA ·
perf - Bounded replay for YOCO consumer layers (replay only the last 140 tokens) ·
perf
Indexer
DeepInfer-Lab 79 ms · official 7.9 ms (DeepSelect top-k)
- Fuse prefill index scoring — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - Select top-k with a dedicated kernel ·
perf
MoE
DeepInfer-Lab 410 ms (experts 252, gate 76, communication 53) · official 52.8 ms (DeepGEMM FP4 GEMM, fused gate, DeepEP all-to-all)
- Route tokens with grouped GEMMs and no host sync ·
perf - Combine routed outputs from local rows and dequantize in place — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - Compute the gate GEMM in bf16 on tensor cores — [MoE] Compute the gate GEMM in bf16 on tensor cores #4 (−55.5 ms measured) ·
perf - Compute the gate on local tokens before the all-gather, then fuse scoring and top-k ·
perf - Run experts on FP4 weights without per-step dequantization (132 ms) ·
perf - All-to-all communication; expert load balancing (EPLB) ·
perf
HC and norms
DeepInfer-Lab 122 ms · official ≈ 10 ms
- Move the Hyper-Connections mixing GEMM off fp32 SIMT cores (
Block.hc_mixes, 28.6 ms) ·perf - Fuse the remaining elementwise ops ·
perf
Runtime
Scheduler, multi-GPU coordination and KV cache.
- Paged blocks and state slots, chunked prefill, preemption, zero-overhead scheduling, deterministic sampling, decode CUDA graphs ·
feature - Idle DP ranks run an empty forward (today the other ranks hang in
all_gather_rows) ·bugfix - DP ranks agree on prefill vs decode every step; remove the
.item()sync inall_gather_rows·bugfix - Correct the
_preemptcomment: recomputation reproduces the token only with a batch-invariant forward ·docs - FP8 / FP4 KV cache in the official format ·
perf - Prefix cache and fork inside the scheduler ·
feature - Cost-model prefill chunking, context parallelism, separate prefill and decode engines ·
feature
Features
- DSpark speculative decoding ·
feature - Structured output ·
feature - End-to-end batch invariance ·
feature - Image input ·
feature
Docs
- Make the English README the default — PR [Docs] Make the English README the default #5 ·
docs
Contributing
- One change per PR, titled
[Module] change, with one label. - Run
pytest -q,python -m compileall -q deepinfer_lab testsandgit diff --check; say so when a PR was not tested on GPU. - Compare performance only against a baseline measured in the same job.
简体中文 · 点击展开
如何阅读
- 两层结构。 本页按模块分类，每一项链接到说明这次改动的 issue，issue 里再链接对应的 PR。已完成的项打勾。
- 标题 先在方括号里写模块，再写改动，例如
[Attention] Read sparse attention keys through row indices。PR 与对应 issue 同名，合并时自动关闭该 issue。 - 标签 表示改动类型，标在每一项的末尾：
perf·feature·bugfix·refactor·docs·ci。 - 从哪里读起
- 第一次了解项目：先看「性能演进」，再看 [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 了解第一轮优化。
- 关注某个模块：读对应小节，条目按完成先后排列。
- 研究某次改动：打开它的 issue，里面有数据、官方做法、具体改动、验证和实测；代码在 PR 里。
性能演进
真实 DeepSeek-V4.1-Flash prefill，4 × B200，EP4 / DP4 / CP1，131,072-token 上下文（126,976 已缓存），每步每卡新增 4,096 token。
| 改动 | Issue | 吞吐（tok/s/GPU） | 每步（ms） | 占官方 |
|---|---|---|---|---|
| 初始版本 | — | 1,084.8 | 3,775.9 | 3.0% |
| tensor-core 稀疏注意力 | #1 | 2,571.7 | 1,592.7 | 7.1% |
| 融合 indexer 打分 | #1 | 3,373.1 | 1,214.3 | 9.3% |
MoE 只合并本地行（当前 main） |
#1 | 4,315.9 | 949.0 | 11.9% |
| bf16 gate 矩阵乘（PR 未合并） | #4 | 4,589.4 | 892.2 | 12.7% |
| 按索引读 key（PR 未合并） | #7 | 5,004.4 | 817.7 | 13.8% |
| 官方 DeepInfer | — | 36,170.9 | 113.2 | 100% |
两行未合并的数据在同一作业内与 main 对照测得（该作业中 main 为 4,322.1 tok/s/GPU），两者合并后的效果尚未测量。之前各行来自不同作业（#1）。官方数据来自更早的作业，占比仅供参考。
Attention
DeepInfer-Lab 每步 281 ms（拼 key 182 + kernel 99）· 官方 14.9 ms（FlashMLA 按索引读分页 KV）
- 稀疏注意力改用 tensor core — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - 稀疏注意力按索引直接读 key — [Attention] Read sparse attention keys through row indices #7（实测 −130 ms）·
perf - 先 profile 按索引读取的 kernel，再缩小与 FlashMLA 的差距 ·
perf - YOCO consumer 层的 bounded replay（只重放最后 140 个 token）·
perf
Indexer
DeepInfer-Lab 79 ms · 官方 7.9 ms（DeepSelect top-k）
- 融合 prefill indexer 打分 — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - top-k 改用专用 kernel ·
perf
MoE
DeepInfer-Lab 410 ms（专家 252、gate 76、通信 53）· 官方 52.8 ms（DeepGEMM FP4 矩阵乘、融合 gate、DeepEP all-to-all）
- grouped GEMM 路由，无 host 同步 ·
perf - 只合并本地行的专家输出，原地反量化 — [Prefill] Sparse attention, indexer and MoE: 1,085 → 4,316 tok/s/GPU #1 ·
perf - gate 矩阵乘改为 bf16 tensor core — [MoE] Compute the gate GEMM in bf16 on tensor cores #4（实测 −55.5 ms）·
perf - 在本地 token 上先算 gate 再 all-gather，并融合打分与 top-k ·
perf - 专家直接用 FP4 权重计算，去掉每步反量化（132 ms）·
perf - all-to-all 通信；专家负载均衡（EPLB）·
perf
HC 与 norm
DeepInfer-Lab 122 ms · 官方约 10 ms
- Hyper-Connections 混合系数的矩阵乘不再走 fp32 SIMT（
Block.hc_mixes，28.6 ms）·perf - 融合其余逐元素算子 ·
perf
Runtime
调度、多卡协调和 KV cache。
- 分页 block 与 state slot、chunked prefill、抢占、零开销调度、确定性采样、decode CUDA graph ·
feature - 空闲的 DP 卡执行空 forward（目前其他卡会卡在
all_gather_rows）·bugfix - 各 DP 卡每步统一 prefill 或 decode；去掉
all_gather_rows中的.item()同步 ·bugfix - 修正
_preempt注释：只有 forward 是 batch-invariant 时，重算才能得到相同 token ·docs - KV cache 改为官方格式的 FP8 / FP4 ·
perf - 调度器内的 prefix cache 与 fork ·
feature - 按代价模型切分 prefill chunk、context parallel、prefill 与 decode 分开部署 ·
feature
Features
- DSpark 投机解码 ·
feature - 结构化输出 ·
feature - 端到端 batch-invariant ·
feature - 图像输入 ·
feature
Docs
- 默认 README 改为英文 — PR [Docs] Make the English README the default #5 ·
docs
提交约定
- 一个 PR 只做一项改动，标题写成
[模块] 改动，打一个标签。 - 运行
pytest -q、python -m compileall -q deepinfer_lab tests、git diff --check；没有在 GPU 上测过的 PR 要写明。 - 性能只与同一作业内测得的基线比较。
Activity
[-][Roadmap] mini-DeepInfer 路线图（2026 Q4）：对齐官方 DeepInfer[/-][+][Roadmap] mini-DeepInfer 2026 Q4: closing the gap to the official DeepInfer / 路线图[/+][-][Roadmap] mini-DeepInfer 2026 Q4: closing the gap to the official DeepInfer / 路线图[/-][+][Roadmap] 2026 Q4: close the gap to official DeepInfer[/+]