Author: The Humanoid Intelligence Team
OpenMOSS Group / Shanghai Innovation Institute / Fudan University
English | 简体中文 · Docs: Model families · API reference · Changelog · Contributing
![]() LeRobot | ![]() openpi | ![]() GR00T | ![]() StarVLA | ![]() XPolicyLab | ![]() RLinf | ![]() PokeBot |
![]() LIBERO | ![]() LIBERO | ![]() RoboCasa GR-1 | ![]() RoboCasa GR-1 | ![]() RoboCasa GR-1 | ![]() LIBERO | ![]() LIBERO |
Contents
hiVLA serves vision-language-action (VLA) models over the network. You point it
at a checkpoint, start hivla serve, and a robot-side caller sends observations
(joint state, camera images, an instruction) and gets back a chunk of actions to
execute. Everything model-specific runs on the server: resizing, normalization,
tokenization, the forward pass, unnormalization, and splitting the output into
the action fields the robot understands.
- One interface for many model families. LeRobot, openpi (π₀ / π₀.₅),
GR00T, StarVLA, XPolicyLab and RLinf models sit behind the same request and
reply format, so the caller does not change when the model does. A
genericfamily serves plain torch / ONNX / TensorRT checkpoints, and a new family is one adapter class. - One environment per model, started for you. Each family keeps its own
torch / jax versions in its own environment;
hivla launchstarts the server with that environment's Python from wherever you are. - Fits hiRobot's interface for many robots. Requests and replies use
hiRobot's
ObservationandActionChunkfields, so a model and a robot meet through one interface. One VLA can serve several robots, with one config per robot that sets its state layout and action width: XPolicyLab'sdemo_policyships configs for the Unitree G1 and dual UR5e arms. One robot can call several VLAs: the Fourier GR-1 in RoboCasa has run both GR00T N1.6 and StarVLA QwenGR00T through hiRobot. - Faster serving. Dynamic batching of concurrent requests, a cache that
replays the last chunk for a repeated observation,
torch.compile, and an async engine that predicts the next chunk while the robot executes the current one. - Many GPUs behind one address.
hivla routeload-balances a pool of servers (round-robin, least-outstanding, or consistent-hash for cache hits); a dead backend returns an error at once instead of hanging the caller.
hiVLA serves models. With hiRobot
installed, hivla.robot.run_episode also runs a served model on a robot, in
simulation or on hardware; it is a minimal loop, and the full agent loop lives
in hiAgent.
Note
Companion projects.
hiRobot is the robot side:
one interface for different robots, which reads observations and checks every
command. hiVLA's requests and replies use hiRobot's Observation and
ActionChunk field names; only hivla.robot imports hiRobot, when it is called.
hiWBC handles whole-body
balance and walking below hiRobot.
hiAgent, the agentic
operating layer, calls hiVLA through hivla.client inside its own loop and
plays the returned chunks on hiRobot.
Warning
Status: early development. APIs are unstable and may change.
Safety on real robots. The action chunks hiVLA serves reach a robot through
hiRobot (from hivla.robot or hiAgent), and a robot under software control
can injure people and damage itself. Read this repository carefully before running it on hardware,
and hiRobot's setup guide
for your robot.
Debug in simulation until it works, then move to the real robot step by step:
read-only first, then the robot hung on a gantry at low gains, with an e-stop
within reach and nobody inside the robot's reach.
Each family runs the model through its own code and pre- / post-processing, so results match the original release. hiVLA adds a common entry point around it.
| Family | Models | Built-in config | Verified | Guide |
|---|---|---|---|---|
generic |
Any torch / ONNX / TensorRT checkpoint | example |
generic | |
lerobot |
LeRobot v0.6.2 single-observation policies such as ACT | lerobot_act_aloha |
ACT ALOHA: chunk shapes over HTTP / WS | lerobot |
openpi |
π₀, π₀.₅, π₀-FAST from Physical Intelligence | openpi_pi05_libero |
π₀.₅ LIBERO: 97.05% mean over 4 suites (openpi: 96.85%) | openpi |
gr00t |
NVIDIA GR00T N1.6 / N1.7 | gr00t_n1d6_gr1 |
N1.6 RoboCasa GR-1: 5 / 10 episodes | gr00t |
starvla |
StarVLA baseframework checkpoints (QwenGR00T, QwenOFT, ...) |
starvla_qwengr00t_gr1 |
QwenGR00T RoboCasa GR-1: 37 / 50 (StarVLA: 0.58–0.74) | starvla |
xpolicylab |
Any XPolicyLab policy (stateful) | xpolicylab_demo_g1 |
demo_policy on the MuJoCo G1 |
xpolicylab |
rlinf |
π₀ / π₀.₅ in RLinf's PyTorch reimplementation | rlinf_pi05_libero |
Pi05-LIBERO-SFT ran LIBERO spatial | rlinf |
pokebot |
Pokebot's cloud RoboAPI model poke-aurora (stateful, no local weights) |
pokebot_aurora |
One real turn: a (40, 8) pose block in 1.7 s | pokebot |
Each guide covers the environment, the config keys, how observation fields reach the model and how its output comes back, and common pitfalls. The family index has the general workflow.
The echo backend returns a zero action chunk, so you can try the server and
the client with no checkpoint, GPU, or deep-learning stack:
git clone https://github.com/humanoidintelligence/hiVLA.git && cd hiVLA
pip install -e ".[server,client]"
hivla serve --config echo_example # HTTP + WebSocket on :8000From another shell:
curl localhost:8000/health # {"status":"ok"}
curl localhost:8000/metadata # what the model expects and returnsimport numpy as np
from hivla.client import VlaClient
client = VlaClient("http://localhost:8000")
client.reset() # fetches /metadata into client.metadata
chunk = client.act({
"joint_pos": np.zeros(7, np.float32),
"images": {"head": np.zeros((224, 224, 3), np.uint8)},
"instruction": "pick up the cube",
})
print(chunk["mode"], chunk["joint_targets"].shape) # position (16, 7)VlaWsClient has the same methods over one persistent WebSocket connection.
reset() also starts an episode on the server, which a stateful model
(XPolicyLab) needs before the first act.
The observation fields and the chunk fields are listed in the
API reference.
Each model family pins its own torch / transformers versions, and they conflict with each other. So hiVLA does not bring a deep-learning stack: you install it as a light library into the model's own environment, and run one server per model. Pick the extras you need:
| Extra | Adds |
|---|---|
server |
hivla serve and hivla route (FastAPI, uvicorn) |
client |
hivla.client and the remote backend (httpx, websockets) |
torch / onnx / tensorrt |
The generic family's backends |
vision |
Image resizing with Pillow |
dev |
pytest and ruff |
The server does not have to be started from the model's environment.
hivla launch reads launch.env from the config (a conda env name, an env
prefix or a Python path), checks that hiVLA imports there and the port is free,
starts hivla serve with that Python and waits until /health answers:
launch:
env: gr00tn1d6 # the model's environment
ready_timeout: 600 # seconds to wait for /healthhivla launch --config <model> # from any env with hiVLAEach family's environment and checkpoint are described in its guide.
This serves NVIDIA's GR00T N1.6-3B for the Fourier GR-1 humanoid (embodiment
tag gr1), starting from a fresh clone.
1. Install into a GR00T environment. Set up
Isaac-GR00T n1.6 by its instructions
(Python 3.12, torch 2.7.1+cu128), then add the Eagle backbone's flash-attn and
hiVLA into that same environment:
git clone https://github.com/humanoidintelligence/hiVLA.git && cd hiVLA
conda activate gr00tn1d6 # your GR00T environment
pip install flash-attn --no-build-isolation # the Eagle backbone needs it
pip install -e ".[server,client]"2. Download the checkpoint from Hugging Face. GR00T's VLM backbone is gated,
so first request access to
nvidia/GR00T-N1.6-3B and log in
with hf auth login:
export HIVLA_MODELS=/path/to/your/models
hf download nvidia/GR00T-N1.6-3B --local-dir $HIVLA_MODELS/Isaac-GR00T-n1.6/checkpoints/GR00T-N1.6-3Bsrc/hivla/configs/gr00t_n1d6_gr1.yaml reads the checkpoint path from ${HIVLA_MODELS}.
Configs expand ~, $VAR, ${VAR} and ${VAR:-default}, so they hold no
machine-specific paths. An unset variable without a default is an error, so a
path can't silently point at the wrong place.
3. Check the config, then serve. hivla inspect resolves the config without
loading weights:
hivla inspect --config gr00t_n1d6_gr1 # prints what will be served
hivla serve --config gr00t_n1d6_gr1 # loads GR00T on the GPU, listens on :80004. Check that it answers, from another shell:
curl localhost:8000/health # {"status":"ok"}
curl localhost:8000/metadata # action_dim 29, cameras, action_horizon, ...To test the model without a running server, scripts/smoke_gr00t_n1d6.py
loads it in-process, sends synthetic observations and checks every chunk's
shape (--http / --ws go through a local server it starts itself):
python scripts/smoke_gr00t_n1d6.py --config gr00t_n1d6_gr1 --iters 105. Drive a robot with it, with hivla.robot (next section) or through
hiAgent: set vla.endpoint in a hiAgent robot config to http://<host>:8000.
See hiAgent's quick start.
The other families work the same way, each in its own environment with its own
built-in config (hivla configs lists them); their guides list the
steps. Each config reads its checkpoint from an environment variable you must
set: gr00t and starvla from ${HIVLA_MODELS}, openpi from
${OPENPI_DATA_HOME}, rlinf from ${RLINF_CKPT} (and ${OPENPI_DATA_HOME}
for its tokenizer).
hivla.robot.run_episode runs a served model on a hiRobot Robot. The robot's
environment needs hiRobot and hivla[client], none of the model's packages:
from hirobot import make_robot
from hivla.client import VlaWsClient
from hivla.robot import run_episode
robot = make_robot("unitree_g1_mujoco", sim=True)
result = run_episode(robot, VlaWsClient("http://127.0.0.1:8012"), max_steps=100,
instruction="wave")
print(result.steps, result.chunks, result.stopped_by) # 100 50 max_stepsIt resets the robot and the server, asks for a chunk, sends its steps with
send_action, and asks again after the server's n_action_steps. For a
model that reads every executed step (observe: every_step in /metadata) it
sends an observe before each further step. A real robot is paced at its
control_hz; a simulator steps as fast as it runs. On an exception the robot
is told to hold() before the error is raised.
To try it with no weights, serve XPolicyLab's demo_policy for the MuJoCo G1
(src/hivla/configs/xpolicylab_demo_g1.yaml, see the XPolicyLab guide)
and run the code above from hiRobot's environment.
The caller sends a plain observation and gets back a ready-to-run action chunk.
All model-specific processing happens on the server, the same split openpi and
GR00T use for their own deployments. The caller's client
does not change when you switch models, and it needs only numpy plus an HTTP /
WebSocket library (hivla[client]), none of the model's dependencies.
Read the diagram from the bottom up, the way a request travels:
hivla.client(VlaClientover HTTP,VlaWsClientover WebSocket) encodes the observation and sends it, as JSON (arrays in base64) or msgpack (arrays as raw bytes). Nothing is pickled.Serverreceives it on/actor/ws./metadatatells the caller what the model expects and returns (cameras, state layout, joint names, action horizon, whether it is stateful) and/healthis for probes./resetstarts an episode and/observehands a stateful model one observation without asking for a chunk. Withserver.api_keyset,/act,/resetand/observerequire the key;/wsdoes not check it yet, so keep the port on a trusted network.InferenceEnginereplays the cached chunk if the observation repeats, and groups concurrent requests into one batch. For a stateful model it instead serves one session at a time: the one that calledresetlast (an HTTP client'sX-Hivla-Sessionheader, or one WebSocket connection).AsyncInferenceEngineis the in-process variant that predicts the next chunk in the background.VlaPolicyruns the input transforms, the backend, then the output transforms. It finally checks that the chunk is one hiRobot can execute: a known control mode, consistent shapes, one horizon.ModelBackendruns the model: the family's own pipeline, or forgenerica torch / ONNX / TensorRT session.echoreturns zeros andremoteforwards to another hiVLA server.
A YAML config builds the whole stack: it is parsed into a ModelSpec, the
spec's family picks a FamilyAdapter (which backend and transforms to use),
and build_engine assembles the layers above.
Core concepts:
| Concept | Responsibility |
|---|---|
ModelSpec |
Everything about one served model: family, checkpoint, backend, action horizon, normalization stats, transforms. Built from a YAML config. |
FamilyAdapter |
What is specific to one model family: how to load and run it, and how its keys map to hiRobot's. |
ModelBackend |
Runs one model format (echo / torch / onnx / tensorrt / remote, or a family's own pipeline). Arrays in, arrays out. |
Transform |
One step dict -> dict (rename, normalize, pad, slice, sin/cos encode, resize). Chained into the input and output pipelines. |
VlaPolicy |
Runs input transforms, the backend and output transforms, then checks the chunk. |
InferenceEngine |
Owns the policy while serving: warm-up, the repeat-observation cache, dynamic batching. AsyncInferenceEngine predicts in a background worker. |
Server |
Exposes the engine over HTTP (/act, /reset, /observe, /metadata, /health) and WebSocket (/ws) with one wire format. |
hiVLA/
├── src/hivla/
│ ├── client.py # VlaClient / VlaWsClient (no model dependencies)
│ ├── protocol.py # wire format (JSON + base64, or msgpack)
│ ├── robot.py # run_episode: a hiRobot Robot against a server
│ ├── cli.py # `hivla serve` / `launch` / `inspect` / `route` / `configs`
│ ├── core/
│ │ ├── spec.py # ModelSpec: one served model
│ │ ├── config.py # config by path or built-in name, ${VAR} expansion
│ │ ├── transforms.py # Transform steps
│ │ ├── policy.py # VlaPolicy: input transforms → backend → output transforms → chunk check
│ │ └── build.py # config → ModelSpec → backend → policy → engine
│ ├── serving/
│ │ ├── engine.py # InferenceEngine / AsyncInferenceEngine: cache, batching, sessions
│ │ ├── server.py # FastAPI app: /act, /reset, /observe, /metadata, /health, /ws
│ │ ├── router.py # `hivla route` and its routing policies
│ │ └── launch.py # `hivla launch`: serve a model with its own env's Python
│ ├── runtimes/ # run one checkpoint format (used by `generic`)
│ │ ├── base.py # ModelBackend interface
│ │ ├── echo.py # dependency-free test backend
│ │ ├── torch.py, onnx.py, tensorrt.py
│ │ └── remote.py # forwards to another hiVLA server
│ ├── families/ # one file per model family: its adapter and backend
│ │ ├── base.py # FamilyAdapter interface
│ │ ├── generic.py # hiVLA loads the checkpoint through a runtime
│ │ ├── lerobot.py, openpi.py, gr00t.py, starvla.py, xpolicylab.py, rlinf.py
│ │ ├── pokebot.py # Pokebot's cloud RoboAPI over HTTP
│ │ └── native.py # adapter for any package with an obs -> action factory
│ └── configs/ # built-in configs, one per served checkpoint (`hivla configs`)
├── scripts/
│ ├── smoke_gr00t_n1d6.py, smoke_lerobot_act.py # GPU smoke tests (in-process or over the network)
│ └── fetch_parallel.py # parallel checkpoint downloads
├── tests/ # pytest suite; model families are faked, no GPU needed
├── docs/
│ ├── api.md # API reference
│ └── families/ # one guide per model family
│ # (each with a Chinese twin, *.zh-CN.md)
├── CHANGELOG.md # changes in each release (CHANGELOG.zh-CN.md: Chinese)
└── CONTRIBUTING.md # what each kind of change needs
What changed in each release is in CHANGELOG.md.
These measure two separate things: (A) how fast one server runs a model on
one GPU, and (B) how a pool of servers scales behind hivla route. Both use
a synthetic load generator (no simulator) against GR00T N1.6-3B (gr1) on one
datacenter GPU, with a 29-dim state and one 256×256 frame per request, so the
numbers show the serving stack rather than simulation time.
One hivla serve, 400 requests. Inference dominates (~211 ms per request at
baseline; decoding and encoding take under a millisecond), so what helps is a
faster forward pass and how concurrent requests reach it.
| Variant | Load | Throughput (req/s) | p50 (ms) | p99 (ms) |
|---|---|---|---|---|
| baseline (batch 1) | sequential | 6.7 | 125 | 219 |
torch.compile (DiT head) |
sequential | 8.9 | 90 | 189 |
| concurrency 16, no batching | 16 in flight | 6.6 | 2404 | 2516 |
| concurrency 16, dynamic batching | 16 in flight | 44.4 | 319 | 669 |
| repeat-observation cache (80% hits) | sequential | 29.7 | 1.7 | 219 |
torch.compile on the DiT head gives the largest gain per request: the forward
pass drops from 211 to 62 ms. Under concurrency, dynamic batching raises
throughput from 6.7 to 44 req/s; without it, concurrent requests wait in line
for the model and only add latency. The cache answers a repeated observation in
about 1.7 ms, which helps when a control loop resends the same frame. JSON and
msgpack perform about the same here, because inference takes far longer than
the transfer.
Four hivla serve backends, one per GPU, behind one router, with 32 requests in
flight.
| Routing policy | Throughput (req/s) | p50 (ms) | p99 (ms) |
|---|---|---|---|
| round-robin | 26.0 | 1266 | 1894 |
| least-outstanding | 26.2 | 1234 | 1320 |
| consistent-hash | 24.9 | 1174 | 2684 |
Least-outstanding keeps the tail latency lowest because it sends requests away
from busy backends. Scaling is close to linear: one backend sustains 6.4 req/s
at this load and four sustain 26.0 req/s (4.1×); the router only passes
bytes through. When one backend was killed mid-run, the requests routed to it
got a 502 at once (134 of 600 over 23 s, none hung) while the other three kept
serving.
- Write a config for the checkpoint, starting from
src/hivla/configs/example.yaml: the family, the checkpoint path, the action horizon, and underpolicythe action width, control mode and cameras.hivla inspect --config ...shows how it resolves without loading weights. - Use an existing family if one fits.
genericserves a plain torch / ONNX / TensorRT checkpoint and normalizes with yournorm_stats. The named families run the model's own code and take loader options fromnative_kwargs; their guides list the keys. - Otherwise register a
FamilyAdapter. It returns the backend that runs the model and the input / outputTransformlists that map hiRobot's field names to the model's and back. You do not subclassVlaPolicy. See Model families for an example, and add a guide underdocs/families/modelled on the existing ones. A model that keeps state between requests setsstatefulon its backend (see the XPolicyLab guide).
If the model's inputs or outputs differ from hiRobot's conventions, two config
options cover the common cases. policy.state_layout concatenates observation
fields (joint positions, gripper, end-effector pose) into the state vector the
model was trained on. policy.gripper_range maps the model's gripper values to
hiRobot's gripper opening (0 closed, 1 open).
The tests fake every model family, so they run without a GPU or model weights:
pip install -e ".[dev]"
pytestBefore opening a pull request, read CONTRIBUTING.md: what each kind of change needs and how to report what you ran.
- Peng Li: pengli.ds [at] gmail.com
If you use hiVLA in your research, please cite it:
@misc{hivla,
title = {hiVLA: A Unified Serving Layer for Deploying Diverse VLA Models},
author = {Li, Peng and Zhang, Yifei and Zhang, Haozhe and Qiu, Xipeng},
year = {2026},
url = {https://github.com/humanoidintelligence/hiVLA}
}Apache License 2.0. See LICENSE for the full terms.














