[Klaud Cold] Update dsr1-fp4-b200-dynamo-sglang SGLang image to v0.5.19-cu130-runtime (blocked: recipe-pinned ai-dynamo 0.8.1 incompatible) / 将 dsr1-fp4-b200-dynamo-sglang 的 SGLang 镜像更新至 v0.5.19-cu130-runtime（受阻：配方固定的 ai-dynamo 0.8.1 不兼容）

[Klaud-Cold]

Current status / 当前状态

Status: stopped before any GPU dispatch — confirmed incompatibility between the candidate image and this family's recipe-pinned serving stack. PR closed as draft; branch retained so this exact candidate is not reselected.
Next step (manual, out of Klaud Cold scope): raise the dynamo.version pin used by this family's upstream srt-slurm recipe, or move the family onto checked-in recipes with a compatible dynamo.hash (as the MTP sibling already does), then retry the image refresh.

The one real change in this PR is the master image bump for dsr1-fp4-b200-dynamo-sglang in configs/nvidia-master.yaml: lmsysorg/sglang:v0.5.8.post1-cu130-runtime → lmsysorg/sglang:v0.5.19-cu130-runtime@sha256:710bc11443a7b1807d69803386468101bcfced35f86bf8fe92a8209e05a2f052 (manifest-list digest, same pinning style as the existing v0.5.14-cu130@sha256:… entries). Model, precision, topology, workloads, recipe references, router metadata and eval selection are unchanged; the generated matrix (5 points, 2/6/6/2/6 nodes, evals at conc 128/128/64/2048) differs from base only in image. No perf-changelog.yaml entry was added because targeted validation never passed. Green benchmarks would not prove global checks pass; none were run here.

状态：在任何 GPU 派发之前停止 —— 已确认候选镜像与该家族配方固定的服务栈不兼容。PR 以草稿状态关闭；保留分支以避免再次选中同一候选。
下一步（需人工处理，超出 Klaud Cold 范围）：提高该家族上游 srt-slurm 配方所固定的 dynamo.version，或将该家族迁移到带有兼容 dynamo.hash 的仓库内配方（MTP 兄弟家族已采用此方式），然后重试镜像刷新。

本 PR 唯一的实际改动是 configs/nvidia-master.yaml 中 dsr1-fp4-b200-dynamo-sglang 的主镜像：lmsysorg/sglang:v0.5.8.post1-cu130-runtime → lmsysorg/sglang:v0.5.19-cu130-runtime@sha256:710bc11443a7b1807d69803386468101bcfced35f86bf8fe92a8209e05a2f052（manifest-list 摘要，与现有 v0.5.14-cu130@sha256:… 条目相同的固定方式）。模型、精度、拓扑、负载、配方引用、router 元数据与评测选择均未改变；生成矩阵（5 个点，节点数 2/6/6/2/6，评测并发 128/128/64/2048）与基线仅 image 不同。由于定向验证未通过，未追加 perf-changelog.yaml 条目。基准通过不代表全局检查通过；此处未运行任何基准。

Candidate evidence / 候选证据

• Candidate id 906fff6c74191d2d-fa39a9e32aba886a; base SHA cc53d2341f5ba0374c76c592f95cf72c16bfa900; branch klaud/auto-906fff6c74191d2d-fa39a9e32aba886a; head 8ab3bdfbd4078d03018188fe5470dcdea019a850.
• Public observation (/api/v1/latest-images): dsr1 / b200 / dynamo-sglang / fp4 / spec none / disagg / 8192→1024 / single_turn on lmsysorg/sglang:v0.5.8.post1-cu130-runtime, published 2026-02-11. Release feed (/api/v1/framework-releases): sglang v0.5.19. Review reason: release-string-mismatch.
• Family resolution: configs/nvidia-master.yaml:dsr1-fp4-b200-dynamo-sglang (runner cluster:b200-nscale, configs/runners.yaml maps it only to b200-nscale-slurm_00..09). The MTP variant is the separate key dsr1-fp4-b200-dynamo-sglang-mtp and is untouched.
• Registry: lmsysorg/sglang:v0.5.19-cu130-runtime exists on Docker Hub (pushed 2026-09-04T21:25Z; manifest digest sha256:710bc1…a2f052; linux/amd64 image sha256:c31ecf…5fa476). Same CUDA 13.0 runtime line as the current cu130-runtime pin; B200 (sm_100) is a supported Blackwell target of the cu130 build.
• Capacity: check-capacity --cluster b200-nscale exit 0 before edits and again before branch/PR creation (01:51Z and 01:59Z UTC, 2026-09-09).
• Open-PR recheck immediately before claiming: none of the open PRs touching configs/nvidia-master.yaml or the B200 nscale launchers (Validate vLLM Router on GB200: DEP4, DEP8, and 1P/2D #2549, Validate official vLLM Router with srt-slurm #2543, test: validate upstream-native vLLM Router topologies #2731, perf(agentx): refresh B200 vLLM MTP recipe / 刷新 B200 vLLM MTP 配置 #2646, Add DeepSeek-V4-Pro FP4 B200 Dynamo TensorRT-LLM recipes / 添加 DeepSeek-V4-Pro FP4 B200 Dynamo TensorRT-LLM 配置 #2552, [WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 #2302, Add dsv4f-fp8-b200-vllm: DeepSeek-V4-Flash-0731 FP8 B200 vLLM single-node recipe #2535, feat(config): add GLM-5.2 NVFP4 B200 Dynamo-SGLang AgentX recipes / 新增 GLM-5.2-NVFP4 B200 Dynamo-SGLang AgentX 配方 #2673, feat(glm52-agentx): Add B300 Dynamo+TRT-LLM AgentX recipes #2666, refactor: share AgentX matrix generation / 共用 AgentX 矩阵生成逻辑 #2903) modify this key, its image, recipes/b200-fp4/8k1k.yaml or the srt-slurm pin; no klaud/auto-906fff… branch existed.

Baseline (published 2026-02-11) / 基线（发布于 2026-02-11）

• Queries: GET /api/v1/benchmarks?model=DeepSeek-R1-0528&date=2026-02-11&exact=true filtered to hardware=b200, framework=dynamo-sglang, model=dsr1, precision=fp4, spec_method=none, disagg=true, isl=8192, osl=1024, image=lmsysorg/sglang:v0.5.8.post1-cu130-runtime (11 rows); GET /api/v1/workflow-info?date=2026-02-11; GET /api/v1/evaluations?model=DeepSeek-R1-0528&date=2026-02-11&exact=true.
• Producer for all 11 points: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/21888944497/attempts/1 (main sweep for PR [NV] Add B200 FP4 SGlang STP configs #672, head SHA 46e0b5ff02e26f07738eab5105c3c1da6123e8a9). Curve snapshot id 274 is not a producer id. Frozen for all attempts; the old image was never dispatched.
• Published evals: no evaluation rows dated 2026-02-11 exist for this identity (N/A). The most recent published gsm8k rows with matching topology are from 2026-04-19 (run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/24637493068): 7P/2D DEP conc 2048 em_strict 0.9560; TP4 1P/1D conc 64 em_strict 0.9553; DEP4 1P/1D conc 128 em_strict 0.9598. Rows dated 2026-09-01 for this identity have TP8 prefill topologies from a DSV4 PR sweep (run 33447526958) and were excluded as unmatched.

conc	id	prefill	decode	GPUs P+D	tput/GPU (tok/s)	output tput/GPU (tok/s)	median TTFT (s)	median TPOT (ms)	median E2EL (s)
4	94278	1×TP4 EP1	1×TP8 EP1	4+8	400.7	66.86	0.265	6.94	6.65
8	94280	1×TP4 EP1	1×TP8 EP1	4+8	688.6	116.10	0.327	7.96	7.74
16	94277	1×TP4 EP1	1×TP8 EP1	4+8	1097.1	182.33	0.443	9.93	9.66
64	94279	1×TP4 EP1	1×TP8 EP1	4+8	2507.7	417.26	0.727	17.35	16.73
64	94257	1×DEP4	1×TP8 EP8	4+8	2413.3	401.56	1.732	17.13	17.51
128	94256	1×DEP4	1×TP8 EP8	4+8	3554.7	590.53	3.335	21.97	23.52
8	94308	1×DEP4	5×TP8 EP8	4+40	200.7	24.82	0.538	7.21	7.25
4	94307	2×DEP4	5×TP8 EP8	8+40	99.1	13.23	0.520	6.45	6.57
128	94306	2×DEP4	5×TP8 EP8	8+40	1529.6	203.29	2.298	12.24	13.54
1024	94294	7×DEP4	2×DEP8	28+16	5971.1	1824.64	2.698	29.99	30.30
2048	94295	7×DEP4	2×DEP8	28+16	8185.6	2503.19	7.087	39.79	44.00

Initial attempt / 初次尝试

• Image / commit: lmsysorg/sglang:v0.5.19-cu130-runtime@sha256:710bc1…a2f052 @ 8ab3bdfbd4078d03018188fe5470dcdea019a850.

• Run links: N/A — no e2e-tests.yml run was dispatched (reason below). No GPU time was used.

• Benchmark / eval results: N/A — not run.

• Per-point deltas vs baseline: N/A — not run.

• Diagnosis (pre-dispatch, evidence-backed):

  • This family runs through runners/launch_b200-nscale-compat.sh, which for dynamo-sglang + dsr1 + fp4 clones NVIDIA/srt-slurm at pin a98738de9b2233459b5456e9ed71af09ce893f92 and applies the upstream recipe recipes/b200-fp4/8k1k.yaml (not checked into InferenceX). That recipe sets dynamo.version: 0.8.1 and model.container: dynamo-sglang; the launcher maps that alias to the master image's squash file, so the image bump does reach the containers.
  • srtctl at that pin installs the serving stack at job start with pip install … ai-dynamo-runtime==0.8.1 ai-dynamo==0.8.1 (src/srtctl/core/schema.py, DynamoConfig.get_install_commands). ai-dynamo 0.8.1 (PyPI, 2026-01-23) declares sglang==0.5.6.post2 and was validated in production against the current v0.5.8.post1 image.
  • ai-dynamo 0.8.1's worker entrypoint dynamo.sglang imports, unguarded and at module top level: from sglang.srt.server_args_config_parser import ConfigArgumentMerger (dynamo/sglang/args.py:19), from sglang.srt.tracing import trace (dynamo/sglang/request_handlers/handler_base.py:15) and from sglang.srt.utils import get_local_ip_auto, get_zmq_socket, maybe_wrap_ipv6_address (dynamo/sglang/publisher.py:12); all three modules are imported by dynamo/sglang/main.py.
  • In the shipped sglang==0.5.19 wheel (same version the v0.5.19-cu130-runtime Dockerfile installs), sglang/srt/server_args_config_parser.py no longer exists (moved to sglang/srt/utils/server_args_config_parser.py), the sglang/srt/tracing/ package no longer exists (moved to sglang/srt/observability/trace.py), and maybe_wrap_ipv6_address is not defined anywhere in the package. All three existed in v0.5.8.post1. Every prefill/decode worker would therefore exit with ModuleNotFoundError before loading weights; the recipe's health check (max_attempts raised to 720 × 10 s by the launcher) would then hold the 2- and 6-node allocations for up to two hours per job.
  • The master's router.version: "0.8.0" is display metadata only; no launcher or workflow consumes it, so it cannot change the installed dynamo version.
  • The only repairs are a newer dynamo.version/dynamo.hash in the recipe or a different recipe path/launcher pin. The recipe is external and shared, the master recipe reference must be preserved, and the launcher is shared code, so no repair exists within the Klaud Cold edit scope. The MTP sibling dsr1-fp4-b200-dynamo-sglang-mtp already runs lmsysorg/sglang:v0.5.12.post1 with checked-in recipes and dynamo.hash: 5b4bc1dd70965017a737c71b19db5a0aeaa88727; the same pattern is the natural manual fix.

• Outcome: not dispatched — confirmed incompatibility between the candidate image and the family's recipe-pinned ai-dynamo 0.8.1, deliberately not confirmed by burning B200 nodes on a deterministic import failure. Repairs used: 0/5.

• Next step: manual recipe/launcher change by a maintainer, then a new image-refresh candidate.

• 镜像 / 提交：lmsysorg/sglang:v0.5.19-cu130-runtime@sha256:710bc1…a2f052 @ 8ab3bdfbd4078d03018188fe5470dcdea019a850。

• 运行链接：不适用 —— 未派发任何 e2e-tests.yml 运行（原因见下）。未使用 GPU 时间。

• 基准 / 评测结果：不适用 —— 未运行。

• 逐点相对基线变化：不适用 —— 未运行。

• 诊断（派发前，有证据支持）：

  • 该家族通过 runners/launch_b200-nscale-compat.sh 运行；对于 dynamo-sglang + dsr1 + fp4，它会以固定提交 a98738de9b2233459b5456e9ed71af09ce893f92 克隆 NVIDIA/srt-slurm，并使用上游配方 recipes/b200-fp4/8k1k.yaml（未纳入 InferenceX 仓库）。该配方设置 dynamo.version: 0.8.1 与 model.container: dynamo-sglang；启动器把该别名映射到主镜像的 squash 文件，因此镜像更新确实会进入容器。
  • 该固定版本的 srtctl 在作业开始时用 pip install … ai-dynamo-runtime==0.8.1 ai-dynamo==0.8.1 安装服务栈（src/srtctl/core/schema.py 的 DynamoConfig.get_install_commands）。ai-dynamo 0.8.1（PyPI，2026-01-23）声明依赖 sglang==0.5.6.post2，并在生产中已与当前 v0.5.8.post1 镜像验证通过。
  • ai-dynamo 0.8.1 的 worker 入口 dynamo.sglang 在模块顶层无保护地导入：from sglang.srt.server_args_config_parser import ConfigArgumentMerger（dynamo/sglang/args.py:19）、from sglang.srt.tracing import trace（dynamo/sglang/request_handlers/handler_base.py:15）以及 from sglang.srt.utils import get_local_ip_auto, get_zmq_socket, maybe_wrap_ipv6_address（dynamo/sglang/publisher.py:12）；这三个模块均由 dynamo/sglang/main.py 导入。
  • 在实际发布的 sglang==0.5.19 wheel（与 v0.5.19-cu130-runtime Dockerfile 安装的版本一致）中，sglang/srt/server_args_config_parser.py 已不存在（迁移至 sglang/srt/utils/server_args_config_parser.py），sglang/srt/tracing/ 包已不存在（迁移至 sglang/srt/observability/trace.py），且 maybe_wrap_ipv6_address 在整个包中均无定义。三者在 v0.5.8.post1 中都存在。因此每个 prefill/decode worker 都会在加载权重之前因 ModuleNotFoundError 退出；配方的健康检查（启动器将 max_attempts 提高到 720 × 10 秒）随后会让 2 节点与 6 节点分配每个作业最多占用两小时。
  • 主配置中的 router.version: "0.8.0" 仅为展示元数据；没有启动器或工作流消费它，因此无法改变实际安装的 dynamo 版本。
  • 唯一的修复方式是在配方中使用更新的 dynamo.version/dynamo.hash，或更换配方路径 / 启动器固定提交。该配方位于外部且共享，主配置的配方引用必须保留，启动器属于共享代码，因此 Klaud Cold 的编辑范围内不存在可行修复。MTP 兄弟家族 dsr1-fp4-b200-dynamo-sglang-mtp 已使用仓库内配方与 dynamo.hash: 5b4bc1dd70965017a737c71b19db5a0aeaa88727 运行 lmsysorg/sglang:v0.5.12.post1；相同模式是自然的人工修复方案。

• 结果：未派发 —— 已确认候选镜像与该家族配方固定的 ai-dynamo 0.8.1 不兼容，有意不通过占用 B200 节点来复现一个确定性的导入失败。已使用修复次数：0/5。

• 下一步：由维护者人工修改配方 / 启动器，然后产生新的镜像刷新候选。

Final full sweep / 最终全量 sweep

• N/A — never reached: no targeted attempt passed, so no perf-changelog.yaml entry was appended, the PR was never marked ready and full-sweep-enabled was never applied. No run-sweep.yml run exists for this head.
• 不适用 —— 未到达该阶段：没有定向尝试通过，因此未追加 perf-changelog.yaml 条目，PR 未转为 ready，也未添加 full-sweep-enabled 标签。该 head 不存在任何 run-sweep.yml 运行。

Disposition / 处置

• Stop reason: confirmed image incompatibility with the family's fixed serving stack; no in-scope repair. Repairs used 0/5. Owned runs: none dispatched, so nothing to cancel. PR closed as draft with no sweep labels; remote branch klaud/auto-906fff6c74191d2d-fa39a9e32aba886a retained so only this exact candidate is blocked. A changed source image or release (or a fixed recipe/launcher pin) can be selected again.
• 停止原因：已确认镜像与该家族固定服务栈不兼容；范围内无可行修复。已使用修复次数 0/5。自有运行：未派发，无需取消。PR 以草稿状态关闭且无 sweep 标签；保留远程分支 klaud/auto-906fff6c74191d2d-fa39a9e32aba886a，仅阻止这一确切候选。源镜像 / 发布版本变化（或配方 / 启动器固定提交修复后）可再次被选中。

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[Klaud-Cold]

Status: terminated before any GPU dispatch; PR closed as draft, branch retained.

Confirmed finding: this family's serving stack is installed at job start by srt-slurm pin a98738de from the upstream recipe recipes/b200-fp4/8k1k.yaml (dynamo.version: 0.8.1). ai-dynamo 0.8.1 hard-imports sglang.srt.server_args_config_parser, sglang.srt.tracing.trace and sglang.srt.utils.maybe_wrap_ipv6_address, none of which exist in the shipped sglang==0.5.19 wheel (the first two moved, the third was removed), so every worker would fail with ModuleNotFoundError before loading weights.

Action / next step: no e2e-tests.yml run was dispatched and no repair budget was used (0/5), because the only fixes (a newer dynamo pin in the external recipe, or re-pointing the family to checked-in recipes and adjusting the shared launcher) are outside Klaud Cold's edit scope. A maintainer can reuse the MTP sibling's checked-in-recipe + dynamo.hash pattern, then a fresh candidate can be selected. Details and the frozen 2026-02-11 baseline are in the PR body.

Links: image commit 8ab3bdf; baseline producer https://github.com/SemiAnalysisAI/InferenceX/actions/runs/21888944497/attempts/1.

---

状态： 在任何 GPU 派发之前终止；PR 以草稿状态关闭，分支保留。

已确认的发现： 该家族的服务栈由 srt-slurm 固定提交 a98738de 在作业开始时按上游配方 recipes/b200-fp4/8k1k.yaml（dynamo.version: 0.8.1）安装。ai-dynamo 0.8.1 硬性导入 sglang.srt.server_args_config_parser、sglang.srt.tracing.trace 与 sglang.srt.utils.maybe_wrap_ipv6_address，而这三者在实际发布的 sglang==0.5.19 wheel 中均不存在（前两者已迁移，第三者已移除），因此所有 worker 都会在加载权重前因 ModuleNotFoundError 失败。

操作 / 下一步： 未派发任何 e2e-tests.yml 运行，未使用修复预算（0/5），因为唯一的修复方式（在外部配方中使用更新的 dynamo 固定版本，或将该家族改为仓库内配方并调整共享启动器）超出 Klaud Cold 的编辑范围。维护者可复用 MTP 兄弟家族的“仓库内配方 + dynamo.hash”模式，之后可再次选取新的候选。详情与冻结的 2026-02-11 基线见 PR 正文。

链接： 镜像提交 8ab3bdfbd4078d03018188fe5470dcdea019a850；基线生产运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/21888944497/attempts/1。