Conversation
A CUDA build failed at link time on CUDA 12.x: nvlink fatal : Cicc option values for '-ftz' do not match vla_core nvlinks the relocatable device code of bitvla_cuda_kernels and vla_cuda_ops together, but only bitvla compiles with --use_fast_math, which implies -ftz=true. nvlink rejects the mixed inputs. Setting CUDA_RESOLVE_DEVICE_SYMBOLS on both libraries makes each resolve its own device symbols inside its archive, with its own flags, and removes the combined link from vla_core. Relocatable device code stays enabled. Verified numerically neutral: tests/vla_predict_check on a GR00T N1.7 checkpoint gives byte-identical output with separable compilation on and off.
GR00T N1.7 finetunes can declare an action modality RELATIVE, meaning the
model predicts an offset from the observed state rather than an absolute
joint target. Every step of the chunk is an offset from the same reference,
the state of the observation that produced it, so the caller reconstructs
target[t] = state_t0 + delta[t]
The client had no such path and treated the chunk as absolute. On an ALOHA
right-arm checkpoint that put the first commanded pose 1.52 rad from where the
arm actually was; with the reconstruction it sits 0.06 rad away.
Those offsets carry their own per-chunk-step statistics in
meta/relative_stats.json, which ships separately from dataset_statistics.json.
The new --rel-stats-json feeds them in. A modality is treated as relative iff
it appears in that file, which is exactly the set GR00T writes there.
Un-normalise those offsets with min/max, never q01/q99, even when the
checkpoint sets use_percentiles. GR00T swaps the whole norm_params entry for a
relative key with the raw relative_stats dict, which bypasses the branch that
would otherwise substitute the percentiles. Using q01/q99 scales every offset
down by ~2x on average and 3x on wrist_rotate, so the arm covers a fraction of
the intended motion, never arrives, and the next chunk repeats the same
command: a steady one-way drift of 0.05 rad/s that walked wrist_rotate to
-1.83 rad, past the -1.25 the training data ever saw.
Measured against ground-truth actions over 12 frames from both source
datasets, mean |pred - gt| falls from 0.065 to 0.027 rad; wrist_rotate, the
worst dimension, from 0.093 to 0.038.
Also drop the hardcoded EEF modality names. The layout now comes from the
checkpoint's own statistics, so joint-space embodiments (single_arm, gripper)
work alongside the EEF ones. Checkpoints that ship the full EEF set keep the
historical order and decode bit-identically to before.
scripts/gen_gr00t_relative_stats.py regenerates the statistics from the source
LeRobot datasets, mirroring gr00t/data/stats.py::load_relative_actions.
The converter now reports use_relative_action from the per-modality rep of the
finetuned embodiment. The global flag in experiment_cfg is set on checkpoints
whose modalities are all ABSOLUTE too, so reading it alone marks every
checkpoint relative.
Three things kept the node from running against a right-arm policy. The camera topics were wrong. The node subscribed to image_raw, but the RealSense driver on the ALOHA publishes image_rect_raw; nothing named image_raw exists on the graph, so no frame ever arrived and the node sat in "Waiting for data" forever. --front-topic and --wrist-topic now carry the working defaults and can be overridden. The arm was hardcoded. Joint states, the wrist camera and the joint commands all pointed at follower_left, so a right-arm checkpoint read the wrong arm and would have driven the wrong one. --arm-side picks the follower; it defaults to left, so existing setups are unchanged. The observation carried only the EEF-style state names. It now also publishes state.single_arm, the same six numbers grouped the way a joint-space checkpoint declares them, and the client takes whichever its statistics name. Also pass --rel-stats-json through, and throttle the "Waiting for data" line to once a second: the control loop retries at full rate, so an unthrottled log buried every other message under thousands of identical lines in the same millisecond.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Run GR00T N1.7 checkpoints trained on relative actions correctly, and make the ALOHA client usable with a right-arm policy.
meta/relative_stats.jsonis decoded as an offset from the observed state,target[t] = state_t0 + delta[t], with per-chunk-step statistics passed through the new--rel-stats-json.min/max, notq01/q99, even when the checkpoint setsuse_percentiles.x, y, z, roll, pitch, yaw, gripper). Joint-space embodiments (single_arm,gripper) now work. Checkpoints that ship the full EEF set keep the old order and decode bit-identically.scripts/gen_gr00t_relative_stats.pyregeneratesrelative_stats.jsonfrom the source LeRobot datasets, mirroringgr00t/data/stats.py::load_relative_actions.use_relative_actionfrom the per-modalityrepof the finetuned embodiment.--arm-side left|right,--front-topic,--wrist-topic,--state-topic. Camera defaults are nowimage_rect_raw. The "Waiting for data" log is throttled to once a second.CUDA_RESOLVE_DEVICE_SYMBOLSon both CUDA libraries so each device-links inside its own archive.Why
Relative actions were treated as absolute. The engine returns the raw chunk and the client decoded it as joint targets. On an ALOHA right-arm checkpoint the first commanded pose landed 1.52 rad from the arm's real position, a violent jerk on hardware.
q01/q99 under-scales the offsets.
StateActionProcessor.set_statisticsreplaces the wholenorm_paramsentry for a relative key with the rawrelative_statsdict. That runs after theuse_percentilesbranch, so relative keys always normalise withmin/max. Usingq01/q99shrinks every offset by about 2x on average and 3x onwrist_rotate. The arm covers a fraction of each motion, the next chunk repeats the same command, and the joint drifts one way. A trace on the robot showedwrist_rotatewalking at -0.053 rad/s to -1.83 rad, past the -1.25 rad the training data ever reached, while every joint tracked its command with correlation +0.9996.The converter read the wrong flag.
use_relative_actionlives inexperiment_cfg/, notconfig.json, so relative checkpoints converted as absolute. The global flag is alsotrueon checkpoints whose modalities are all ABSOLUTE, such as LIBERO-4suite, so the per-modalityrepis the reliable signal.The ALOHA client could not see the cameras. No topic named
image_rawexists on the robot, so the node waited forever. Every topic also pointed atfollower_left.The CUDA build failed on CUDA 12.8.
vla_corenvlinked both libraries together. Onlybitvla_cuda_kernelsbuilds with--use_fast_math, and nvlink rejects the mix withCicc option values for '-ftz' do not match.Verified
-Wall -Wextra(first-party code). CUDA build on CUDA 12.8, 0 first-party warnings.ctestpasses, 8/8.vla_predict_checkdiff). The engine is untouched. The CMake change gives byte-identicalvla_predict_checkoutput with separable compilation on and off.Decoding checked against ground-truth actions, 12 frames from 4 episodes across both source datasets (
aloha_placing_kitchen_lerobot,aloha_placing_drawer_lerobot), mean |pred − gt| in rad:wrist_rotateq01/q99min/max(this PR)Tensor check: every action-head projector in the converted GGUF (
act_dec,act_enc,state_enc,proj_out), including the per-embodiment slices, is bit-identical to the source safetensors.Relative detection in the converter: ALOHA-RightArm, SO101-Multitask and UR10E-CUP report relative. LIBERO-4suite reports absolute.
Not yet done: a closed-loop run on the robot with the
min/maxdecoding.Archs and backends tested: GR00T N1.7 on CUDA (RTX 4070 Ti SUPER, CUDA 12.8) and CPU. ALOHA right-arm follower with ROS 2 Humble.