[Klaud Cold] glm5.2-fp8-mi355x-sglang-agentic-mtp: day-zero GLM-5.2 FP8 SGLang AgentX MTP on MI355X / MI355X 上 GLM-5.2 FP8 SGLang AgentX MTP 首发配方 - #2867
Conversation
1d0b621 to
b03aa7d
Compare
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # passes no chat_template_kwargs (mini-swe-agent doesn't); the heavy | ||
| # thinking burns the shared 75-step budget. Double it, as the GLM-5.2 B200 | ||
| # and MI325X recipes do. | ||
| export SWEBENCH_AGENT_STEP_LIMIT=150 |
There was a problem hiding this comment.
🔴 SWEBENCH_AGENT_STEP_LIMIT=150 is meant to 'double' the mini-swe-agent step budget, but the real default (benchmark_lib.sh:2545, int(os.environ.get("SWEBENCH_AGENT_STEP_LIMIT","250"))) is 250, not 75 as the comment claims — so this override actually cuts the budget by 40% for this recipe's EVAL_ONLY (SWE-bench) runs. Fix: either drop the override so EVAL_ONLY keeps the true 250 default, or set it to a value actually above 250 if more thinking budget is genuinely needed; verify against the real default instead of the stale '75-step' comment copied from other GLM-5.2 recipes. The FP4 MI355X sibling this script claims to be an unchanged 'serve flags' port of has no such override, so eval-mode results between the two 'comparable' precision curves will diverge.
Extended reasoning...
benchmark_lib.sh's _run_swebench_agentic_generation reads SWEBENCH_AGENT_STEP_LIMIT with Python default "250" and writes it into mini-swe-agent's agent.step_limit. The new script unconditionally exports SWEBENCH_AGENT_STEP_LIMIT=150 in the EVAL_ONLY branch (added net-new vs. the FP4 MI355X sibling, which sets nothing and gets 250). Any EVAL_ONLY run of this recipe therefore gives the agent only 150 commands instead of the 250 it would otherwise get, increasing the LimitsExceeded-unsubmitted rate and depressing the resolved-rate score, while the comment frames this as 'doubling' a 75-step budget that does not exist in code — the same wrong-default claim is copied from glm5.2_fp4_b200/b300 and glm5.2_fp8_mi325x, but this is its first appearance on MI355X where the FP4 sibling has no override at all.
Verification: nit. The comment at benchmarks/single_node/agentic/glm5.2_fp8_mi355x_sglang_mtp.sh:277 claims GLM-5.2 thinking "burns the shared 75-step budget. Double it" and line 279 sets export SWEBENCH_AGENT_STEP_LIMIT=150. The real default is 250, not 75: benchmark_lib.sh:2545 reads int(os.environ.get("SWEBENCH_AGENT_STEP_LIMIT", "250")) and utils/evals/EVALS.md:642 documents the default as 250.…
| - "Image lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907 (2026-09-07 ROCm 7.2 mi35x nightly, digest sha256:1a31d1ff5f6b4afee1866d72c0610445c629fa3b0c056f372f843ef839dbd8be; Docker Hub last pushed 2026-09-07T14:35:05Z), the newest tag in the rocm720-mi35x flavor the other MI355X SGLang recipes use. --dsa-prefill-backend/--dsa-decode-backend tilelang, SGLANG_OPT_USE_TOPK_V2 and SGLANG_SIMULATE_ACC_TOKEN_MODE are all present in v0.5.19." | ||
| - "Memory: the ~756 GB block-FP8 checkpoint (141 shards) is ~94.5 GB/rank at TP8 on 288 GB MI355X (MXFP4: ~47 GB/rank), leaving a ~150 GB/rank fp8 KV pool inside the 0.85 static share (MXFP4 TP8: ~182.7 GB/rank). TP8-only on day zero: TP4 would leave ~56 GB/rank for KV, DSA indexer, and EAGLE verification batches, and the MI325X FP8 sibling is TP8-only for the same reason. HiCache ratio 1.5 pins ~1.8 TB across TP8, inside the ~3.0 TB of host DRAM on cluster:mi355x-amds." | ||
| - "Search space, steps of at least 2 with a hard stop at 16: TP8/EP1 GPU-resident KV at conc [1, 2, 4, 10] (mirrors the MXFP4 sibling's TP8 arm exactly) and TP8/EP1 HiCache DRAM offload at conc [8, 12, 16]; dram-utilization 0.8." | ||
| - "runners/launch_mi355x-amds.sh mounts the NFS hf-hub cache (/it-share/hf-hub-cache) for zai-org/GLM-5.2-FP8, following the MiniMax-M3 precedent, so the 756 GB checkpoint is pulled once for the cluster instead of once per node-local NVMe cache. No script-name routing change was needed: the launcher already resolves <model>_<precision>_mi355x_<framework>_mtp.sh." |
There was a problem hiding this comment.
🔴 The new changelog entry ships with pr-link: PRLINK_PLACEHOLDER even though the PR is already open (opened 2026-09-08), violating configuration-procedures.md's rule that the placeholder is only allowed before the PR exists and must be replaced with the real URL immediately after creation. Fix: replace PRLINK_PLACEHOLDER with the actual PR URL (https://github.com/SemiAnalysisAI/InferenceX/pull/) before merge.
Extended reasoning...
docs/configuration-procedures.md:254 states 'Before the PR exists ... permits pr-link: TBD. Replace it with the real URL immediately after creating the PR.' Every other entry in perf-changelog.yaml (e.g. the entry immediately above at line 6923) has a resolved github.com/.../pull/ link; this new entry breaks that append-only invariant by merging with an unresolved placeholder, leaving the changelog unable to trace this recipe back to its originating PR once merged.
Verification: normal. The new appended entry ends with pr-link: PRLINK_PLACEHOLDER (perf-changelog.yaml:6937), added by this PR. Beyond the traceability rule in docs/configuration-procedures.md:254, this token is not a recognized placeholder: utils/validate_perf_changelog.py:24-27 defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. The automated… | nit.…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34173459478 |
…P8 SGLang AgentX MTP on MI355X Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (EAGLE DSA fp8 read-door None guard) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
b03aa7d to
a7422ba
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34255674868 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34255674868 |
1 participant
Summary
Day-zero GLM-5.2 FP8 AgentX (agentic-coding) recipe on MI355X with SGLang native EAGLE/MTP:
glm5.2-fp8-mi355x-sglang-agentic-mtp, the FP8 precision sibling ofglm5.2-fp4-mi355x-sglang-agentic-mtp.zai-org/GLM-5.2-FP8(~756 GB, 141 shards,quant_method=fp8, 128x128 weight blocks). Quantization is auto-detected from the checkpoint, as the ROCm GLM-5.2 FP8 MI325X sibling does.benchmarks/single_node/agentic/glm5.2_fp8_mi355x_sglang_mtp.sh, a port of the validated MXFP4 MI355X script. Serve flags unchanged: fp8_e4m3 KV, tilelang DSA prefill/decode backends,SGLANG_OPT_USE_TOPK_V2=false(top-k v2 JIT cannot build for gfx950), glm47/glm45 parsers, chunked-prefill 32768, mem-fraction-static 0.85, HiCache ratio 1.5 write_through_selective. EAGLE num-steps 5 / topk 1 / 6 draft tokens with acceptance pinned to golden AL 3.61 (K=5, thinking_on), which was measured on this FP8 checkpoint. The MXFP4 script's Mooncake L3 branch is not carried.lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907(verified on Docker Hub, pushed 2026-09-07T14:35:05Z, digestsha256:1a31d1ff5f6b4afee1866d72c0610445c629fa3b0c056f372f843ef839dbd8be), the newest tag in the rocm720-mi35x flavor the other MI355X SGLang recipes use. The DSA backend flags and env vars the script relies on are present in v0.5.19.runners/launch_mi355x-amds.shmounts the NFS hf-hub cache forzai-org/GLM-5.2-FP8(MiniMax-M3 precedent) so the 756 GB pull happens once for the cluster rather than once per node-local NVMe cache. Script-name routing already resolvesglm5.2_fp8_mi355x_sglang_mtp.sh.Validation:
bash -non the script and launcher, both YAMLs parse,generate_sweep_configs.py test-configyields the expected 7 cells, and the launcher's script-name construction resolves to the new file.Recipes touched:
glm5.2-fp8-mi355x-sglang-agentic-mtpTest plan
🤖 Generated with Claude Code
Note
Low Risk
Changes are limited to benchmark scripts, cluster launcher cache routing, and YAML/changelog entries with no production serving or auth paths touched.
Overview
Adds the first GLM-5.2 FP8 AgentX (
agentic-coding) benchmark on MI355X via SGLang EAGLE/MTP, as the FP8 sibling of the existing MXFP4 recipe.A new run script
benchmarks/single_node/agentic/glm5.2_fp8_mi355x_sglang_mtp.shports the validated MXFP4 MI355X agentic flow forzai-org/GLM-5.2-FP8(~756 GB, TP8-only), with HiCache DRAM offload at higher concurrency and throughput runs pinning simulated acceptance to golden AL 3.61. Master configglm5.2-fp8-mi355x-sglang-agentic-mtpdefines a 7-cell sweep (GPU-resident KV at conc 1/2/4/10; HiCache at 8/12/16) onlmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260908(re-pinned after intermittent DSA fp8 KV crashes on the 20260907 image).runners/launch_mi355x-amds.shnow mounts the shared NFS Hugging Face cache for this checkpoint (same pattern as MiniMax-M3).perf-changelog.yamldocuments the recipe.Reviewed by Cursor Bugbot for commit a7422ba. Bugbot is set up for automated code reviews on this repo. Configure here.
Update: re-pinned to the 2026-09-08 ROCm daily
First sweep (run 34173459478) on
v0.5.19-rocm720-mi35x-20260907: the four GPU-resident cells (c1/c2/c4/c10) and HiCache c8 passed; HiCache c12, c16 and the eval crashed after minutes to an hour of clean serving withAttributeError: 'NoneType' object has no attribute 'translate_dcp_read_ids'on the DSA fp8 KV read path (the EAGLE draft backend never binds a translator). sgl-project/sglang#38318 (merged 2026-09-07T20:03Z) adds the None guard. Re-pinned tolmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260908(pushed 2026-09-08T14:36:46Z, digestsha256:976bdf01d2e41ed5354741f99afc2cf0164e4f25607a1263e2f5c7519c60fa9f), the first mi35x daily built after that fix. Recipe otherwise unchanged; the passing cells show the port itself is sound.