dsv4-fp4-b300-sglang-agentic-hicache-mtp: restore MegaMoE FP4 acts, retarget to cluster:b300-dsxe / 恢复 MegaMoE FP4 激活路径，并迁移至 cluster:b300-dsxe - #2878
Conversation
sglang#35918 deprecated the SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS / _USE_MXF4_KIND env vars to warn-only, so the nightly-20260827 image runs MegaMoE on the default FP8-acts path: -8.6% at conc 384 and the conc-128 OOM. Pass the flag, and restore conc 64/128 to the hicache row. sglang#35918 将 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS / _USE_MXF4_KIND 两个环境变量废弃为仅告警，nightly-20260827 镜像因此回退到默认 FP8 激活 路径：conc 384 下降 8.6%，conc 128 OOM。改为显式传 --enable-w4a4-mxfp4-megamoe，并把 conc 64/128 加回 hicache 搜索网格。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
仅把本 PR 的条目指向 #2878，TileRT 条目保留原占位符
|
Claude finished @yhyang201's task in 3m 15s —— View job PR Review — dsv4-fp4-b300-sglang-agentic-hicache-mtp MegaMoE FP4 restore + DSXE retarget
SummaryNo blocking issues found. One 🟡 warning posted inline: the script comment claims the recipe "works on either" build generation, but Everything else checks out:
One caveat: the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # measured. On builds predating sgl-project/sglang#35918 these two env vars | ||
| # are what forwards DG_USE_FP4_ACTS / DG_USE_MXF4_KIND to DeepGEMM; on newer | ||
| # builds they are deprecated no-ops and --enable-w4a4-mxfp4-megamoe (passed | ||
| # above) carries the forwarding. Export both so the recipe works on either. |
There was a problem hiding this comment.
🟡 WARNING: The claim "Export both so the recipe works on either" is not accurate — the recipe does not work on pre-#35918 images.
Why it matters: --enable-w4a4-mxfp4-megamoe is passed unconditionally on the DP-attention path (line 135). On any image predating sgl-project/sglang#35918, argparse will reject it as an unrecognized argument and the server will refuse to start — the env exports never get a chance to matter. If someone rolls the image back (a common move in this repo when a nightly regresses), the DP lane fails at launch, and this comment tells them it's safe.
Fix: Correct the comment so a future rollback isn't attempted on the strength of it:
| # measured. On builds predating sgl-project/sglang#35918 these two env vars | |
| # are what forwards DG_USE_FP4_ACTS / DG_USE_MXF4_KIND to DeepGEMM; on newer | |
| # builds they are deprecated no-ops and --enable-w4a4-mxfp4-megamoe (passed | |
| # above) carries the forwarding. Export both so the recipe works on either. | |
| # measured. On builds predating sgl-project/sglang#35918 these two env vars | |
| # are what forwards DG_USE_FP4_ACTS / DG_USE_MXF4_KIND to DeepGEMM; on newer | |
| # builds they are deprecated no-ops and --enable-w4a4-mxfp4-megamoe (passed | |
| # above) carries the forwarding. NOTE: pre-#35918 images reject that flag at | |
| # argparse time, so rolling the image back also requires dropping the flag; | |
| # these exports alone are not enough. |
(Alternatively, gate the flag on a --help probe of sglang.launch_server, but fixing the comment is enough given the config pins a post-#35918 image.)
There was a problem hiding this comment.
Looks good, straightforward and well-documented change. Reviewed the new --enable-w4a4-mxfp4-megamoe flag and comments in the agentic script (only applies to the DP-attention path pinned to the exact image where the flag exists — no backward-compat parsing risk), the cluster:b300-dsxe retarget (verified this cluster exists in configs/runners.yaml, unlike the now-orphaned cluster:b300-nv), the conc-list expansion against the existing MEM_FRACTION_STATIC tiering logic, and confirmed the perf-changelog.yaml entry is a pure tail-append (no prior bytes touched).
Extended reasoning...
Overview
The diff touches three files for one recipe (dsv4-fp4-b300-sglang-agentic-hicache-mtp): the agentic benchmark script adds --enable-w4a4-mxfp4-megamoe plus updated comments explaining why the old env vars became no-ops after sglang#35918; configs/nvidia-master.yaml retargets the recipe's runner from cluster:b300-nv to cluster:b300-dsxe and expands the DP-attention/HiCache row's conc-list from 5 to 7 entries (adding 64 and 128); perf-changelog.yaml gets one new append-only entry.
Security risks
None. This is a benchmark configuration/script change with no auth, crypto, network-exposure, or user-input-handling implications. No secrets or credentials are introduced.
Level of scrutiny
Low-to-medium is appropriate: config/script tweaks to an internal benchmarking harness, not production or security-critical code. I verified the specific factual claims rather than taking the PR description at face value: (1) confirmed via git log -- configs/runners.yaml that cluster:b300-nv was indeed removed in prior PRs (#2818/#2826) and cluster:b300-dsxe is a valid, currently-defined cluster; (2) confirmed the new --enable-w4a4-mxfp4-megamoe flag is added only inside the DP_ATTENTION=true branch, gated to the exact pinned image, with no other recipe/script sharing this file at an older image (the similarly-named non-agentic script under benchmarks/single_node/ is a distinct file pinned to an older image); (3) confirmed the perf-changelog.yaml diff is a pure append at the physical tail (git diff stat shows only insertions in that file); (4) confirmed the --use-chat-template/run_benchmark_serving invariant noted in the task conventions does not apply to this file, since agentic scripts use run_agentic_replay_and_write_outputs instead.
Other factors
The change is small (24 lines total), self-contained, well-commented, and consistent with the described root cause. No CODEOWNERS concerns are evident, and the PR conversation timeline shows no outstanding third-party objections — just the author's own commits and a bot comment.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34176074502 |
1 participant
Summary / 概述
EN: The DP-attention MegaMoE path has silently run the default FP8-acts JIT kernel since the image bump in #2759, costing -8.6% output throughput at conc 384 and the conc-128 OOM that removed two grid points. sgl-project/sglang#35918 (first shipped in lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1) deprecated SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS / _USE_MXF4_KIND to warn-only no-ops behind the new --enable-w4a4-mxfp4-megamoe flag; the recipe's env exports stopped forwarding DG_USE_FP4_ACTS / DG_USE_MXF4_KIND to DeepGEMM. This PR passes the flag, restores conc 64/128 to the HiCache row (12-point grid), and retargets the lane from the decommissioned cluster:b300-nv (#2818) to cluster:b300-dsxe (#2826).
中文： 自 #2759 换镜像起，DP-attention 的 MegaMoE 一直在静默运行默认的 FP8 激活 JIT kernel，导致 conc 384 输出吞吐下降 8.6%，并引发 conc 128 OOM（两个网格点因此被移出）。sglang#35918（首次随 nightly-dev-cu13-20260827-20621aa1 发布）把 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS / _USE_MXF4_KIND 废弃为仅告警的 no-op，改用 --enable-w4a4-mxfp4-megamoe；配方原有的 env 导出不再向 DeepGEMM 转发 DG_USE_FP4_ACTS / DG_USE_MXF4_KIND。本 PR 传入该 flag、把 conc 64/128 加回 HiCache 行（恢复 12 点网格），并将 lane 从已退役的 cluster:b300-nv（#2818）迁移到 cluster:b300-dsxe（#2826）。
Root cause evidence / 根因证据
EN: Run 33051183882 (FP8-acts) vs run 32695861783 (FP4-acts), identical recipe flags: conc 384 output throughput 5,351 -> 4,890 tok/s; per-decode-step +7.8% at identical batch size, accept length, 100% CUDA-graph coverage and ~95% prefix hit; prefill input throughput -8.2%. The TP8 no-offload row (flashinfer, no MegaMoE) is unchanged within +/-0.3% at c1-c32 on the same two images, and dcgm SM clocks differ by <1%. The new run's server log shows 60 deprecation warnings for the two env vars and enable_w4a4_mxfp4_megamoe: False in server_args. The same fallback explains the conc-128 OOM (FP8 acts double the dispatch workspace; the 0.93 mem-fraction tier's headroom was exhausted).
中文： 对比 run 33051183882（FP8 激活）与 32695861783（FP4 激活），配方 flag 完全一致：conc 384 输出吞吐 5,351 -> 4,890 tok/s；在 batch 大小、accept length、100% CUDA graph 覆盖率、约 95% prefix 命中率均相同的情况下，单 decode step 时间 +7.8%；prefill 输入吞吐 -8.2%。同两个镜像上 TP8 无 offload 行（flashinfer，不走 MegaMoE）在 c1-c32 差异不超过 +/-0.3%，dcgm SM 频率差异 <1%。新 run 的 server log 中有 60 条两个 env 变量的废弃警告，且 server_args 里 enable_w4a4_mxfp4_megamoe 为 False。conc 128 的 OOM 也由同一原因解释（FP8 激活使 dispatch workspace 翻倍，耗尽 0.93 mem-fraction 档位的余量）。
Changes / 改动
EN:
中文：
EN: First run of this key on DSXE hardware; it also serves as the NV->DSXE cluster A/B check. Expected: conc 384 back to ~5,350 tok/s, conc 64/128 measured without OOM.
中文： 这是该 key 在 DSXE 硬件上的首次运行，同时作为 NV->DSXE 集群的 A/B 对照。预期：conc 384 恢复至 ~5,350 tok/s，conc 64/128 不再 OOM 并完成测量。
Note
Low Risk
Benchmark launcher and YAML sweep config only; no application auth or data-path changes, though cluster and concurrency grid changes affect where and how perf jobs run.
Overview
Fixes a silent MegaMoE regression on the DP-attention path after the SGLang image from #2759: deprecated
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS/_USE_MXF4_KINDno longer forward to DeepGEMM, so runs fell back to FP8-acts JIT (~8.6% lower output throughput at conc 384 and conc-128 OOM from doubled dispatch workspace).The agentic recipe now passes
--enable-w4a4-mxfp4-megamoewhile still exporting the old env vars for pre-#35918 images.dsv4-fp4-b300-sglang-agentic-hicache-mtpmoves fromcluster:b300-nvtocluster:b300-dsxeand restores conc 64 and 128 on the DEP8 HiCache sweep (12-point grid).perf-changelog.yamldocuments root cause, A/B numbers, and why higher conc points masked the step-time hit.Reviewed by Cursor Bugbot for commit 3ccb09a. Bugbot is set up for automated code reviews on this repo. Configure here.