Repository navigation
Conversation
|
Can we add some accuracy benchmark instruction or some references of whether this is correct? |
|
Requesting reviews from @nvluxiaoz @qmiao-hub @rickzw so that we can merge this PR. Thanks! |
nvluxiaoz
left a comment
nvluxiaoz
left a comment
There was a problem hiding this comment.
LGTM as long as ever has validated.
| ": \"${OUTPUT_JSON:?run the setup cell first}\"\n", | ||
| "\n", | ||
| "cd \"$EDGELLM_ROOT\"\n", | ||
| "./build/experimental_models/cosmos3/examples/cosmos3_policy_inference \\\n", |
There was a problem hiding this comment.
I think we need to add --viewPoint concat_view here to match the concatenated DROID camera input. @nvluxiaoz to confirm.
Align the TensorRT-Edge-LLM policy notebook with the vLLM-Omni/SGLang observation (640x540 multiview + pick-and-place prompt) instead of the reasoner still and upstream CLI placeholder.
The v0.11.0 policy CLI defaults to ego_view, so the DROID concat input has to pass --viewPoint concat_view.
38e8d06 to
57f4f47
Compare
The overview was saying every backend covers forward dynamics, inverse dynamics, and policy, while TensorRT-Edge-LLM only runs the Edge policy task.
|
Hi @ever-wong , thank you for the comments. Rebased onto current main. The notebooks now pin TensorRT-Edge-LLM v0.11.0, and the policy command passes I built the v0.11.0 Could you confirm whether this cookbook should stay on the VLA-guide export path until that contract matches, or switch to the experimental builder? I have not re-exported or rebuilt engines on v0.11.0 yet. |
…books (#300) ## Summary Moves the Cosmos3-Edge TensorRT-Edge-LLM user guide from NVIDIA/cosmos#326 into this repo, under `.agents/skills/cookbooks/cosmos3/`. - Shared setup in `.agents/skills/cookbooks/cosmos3/README.md`, pinned to TensorRT-Edge-LLM v0.11.0. `EDGELLM_ROOT` defaults to `$HOME/TensorRT-Edge-LLM`. - Reasoner notebook: export `nvidia/Cosmos3-Edge` with `--task reasoning`, build engines under `ENGINE_DIR/reasoning`, and run `llm_inference` on `assets/robot_153.jpg`. - Policy notebook: on v0.11.0, build `und-prefill`, `gen`, and `vae-encoder` with `tensorrt-edgellm-build` under `ENGINE_DIR/policy`, then run `cosmos3_policy_inference`. The C++ runtime needs `-DBUILD_EXPERIMENTAL_MODELS=ON` and `-DENABLE_CUTE_DSL=ALL`. It uses the same 640×540 DROID `concat_view` as the vLLM-Omni / SGLang policy notebooks, with `--viewPoint concat_view` and an 8-value `--state`. Edge-LLM covers the policy task only. - Notebooks resolve assets from `.agents/skills/cookbooks/cosmos3`. On the v0.11.0 tag, `tensorrt-edgellm-export --task policy` writes the UND input `attention_pos_id` and VAE `height` / `width` only. `cosmos3_policy_inference` binds `attention_position_ids` and reads `builder_config.content_height` / `content_width`. The direct builder writes that contract, so the policy cookbook uses `tensorrt-edgellm-build`. Upstream references: [Cosmos3-Edge VLA guide](https://nvidia.github.io/TensorRT-Edge-LLM/user_guide/examples/vla/cosmos3.html) for reasoning, and the [direct engine builder](https://nvidia.github.io/TensorRT-Edge-LLM/user_guide/getting_started/direct-engine-builder.html) for policy. ## Testing - Asset paths resolve from both notebook directories: `robot_153.jpg` and the three DROID camera videos are found. - Policy, pinned v0.11.0 (`95515c2`), x86 RTX PRO 6000 (SM120): `tensorrt-edgellm-build --components und-prefill,gen,vae-encoder`, then `cosmos3_policy_inference --viewPoint concat_view --state 0,0,0,0,0,0,0,0`. `action.json` is `[1, 32, 8]`, `finite=true`. The zero state is a runtime smoke, not a DROID quality check. v0.11.0 `cosmos3_policy_inference --help` lists `concat_view`. - Reasoner was not re-run on v0.11.0. The last full export → `llm_build` + `visual_build` → `llm_inference` sequence was TensorRT-Edge-LLM v0.10.0 on the same RTX PRO 6000. After Ever's visual-export fix, `llm_inference` was re-run on that box for `robot_153.jpg` and `describe_anything.png`, and the captions matched the images. Ever confirmed normal outputs on Thor. See the thread on NVIDIA/cosmos#326. That retest did not name the tag as v0.11.0. --------- Co-authored-by: pengcuo <pzeren@nvidia.com>
Summary
cookbooks/cosmos3/README.md. Policy needs-DBUILD_EXPERIMENTAL_MODELS=ON.Testing
Tested on an x86 developer GPU (RTX PRO 6000, SM120) with TensorRT-Edge-LLM v0.10.0. Both added notebooks in this PR were run top-to-bottom (
run_with_trt_edge_llm.ipynbandrun_policy_with_trt_edge_llm.ipynb).--task reasoning→llm_build+visual_build→llm_inferenceonrobot_153.jpg. Pipeline finished.--task policy→cosmos3_policy_build→cosmos3_policy_inference.[1,16,10],finite=true, ~195 ms (runtime smoke).On this SM120 box the pipeline succeeded for both notebooks (export, engine build, and C++ inference finished; policy returned
[1,16,10]withfinite=true). Reasoner captions were not usable (text did not matchrobot_153.jpg); policy was only checked as a finite action tensor, not DROID quality.Likely cause: TensorRT engines and ViT/FMHA kernels are SM-specific. The device used for this testing is RTX PRO 6000. Official Edge-LLM targets are Thor / Spark / Orin, etc. I do not have an Official Edge-LLM device (Jetson Thor / DRIVE Thor / DGX Spark / Orin), so caption or policy quality on those SKUs are not checked. x86 is Developer-only; on this SM120 box reasoner captions were not usable and should not be treated as gold. Re-run is needed on Thor, or Spark) before treating quality as validated.
The above issue has been resolved (please refer to the comment threads below).