2023 2026

Full-Stack Humanoid
Loco-Manipulation

From a robot dog with an arm
to a humanoid that plans its own moves
Qingwei Ben  ·  CUHK MMLab  ·  NVIDIA GEAR
The speaker

Seventeen papers, one goal

The goal
Qingwei Ben
Ph.D. candidate, CUHK MMLab
Research intern, NVIDIA GEAR
“My goal is to develop full-stack algorithms that bring robots into everyday homes.”
www.qingweiben.com
The path
2020 – 2024 · B.Sc. & B.Eng.
Weiyang College, Tsinghua University
Mathematics and Physics + Software Engineering
2023 · Undergraduate researcher
TEA Lab, IIIS, Tsinghua University
2023 · Visiting student
UMass Amherst / MIT-IBM Watson AI Lab
2024 – 2026 · Research intern
Shanghai AI Lab / Intern Robotics
2024 – 2028 · Ph.D. student
The Chinese University of Hong Kong / MMLab
2026 · Research intern
NVIDIA GEAR Lab
So far
17papers
1000+citations
Venues
RSS ×3CVPR ×3ICRA ×2ICLRCoRLRA-LScience Robotics
News
Oct 2026

GRAIL accepted to CoRL 2026 as an Oral

Aug 2026

SONIC accepted by Science Robotics

Jun 2025

HoST: RSS 2025 Outstanding Systems Paper Award finalist

Source: www.qingweiben.com and the CV linked there, as of 19 Nov 2026 · 17 papers: our count of the CV’s lists · venue counts: CV · citations: homepage · HoST award name as listed by RSS 2025
StartRootsMoveHandsSystem 0SeeSystem 1
02
The hook

Same robot, a different driver

2025 · HOMIE · a person in a cockpit drives a Unitree G1 (1×)
2026 · GR00T 2 Turbo · two Unitree G1s scoop and serve popcorn (autonomous, 1×)
every command written by a person
every command written by a model on the robot
Both are a Unitree G1 doing whole-body loco-manipulation. This talk is about everything that had to be built to take the person out of the loop.
Clips: HOMIE project page (RSS 2025), normal speed · GR00T 2 project page (NVIDIA whitepaper, Nov 2026), excerpt of an uncut autonomous rollout at 1×
StartRootsMoveHandsSystem 0SeeSystem 1
03
Motivation

Why not one big network?

A humanoid has to decide at three speeds at once. In the 2026 stack, no single network does all three.
2025 · SONIC · a swinging weight hits a real G1
01

Balance cannot wait

The body must stay balanced through every motion. Joint PD control runs at 1 kHz, fed with new joint targets at 50 Hz.

2026 · GR00T 2 Turbo · a G1 wipes a spill with its foot
02

Seeing and planning take time

One prediction from camera images takes 124 ms on the robot's Jetson AGX Thor. It plans 1.6 s, about once per second.

2026 · GR00T 2 Turbo · trash pickup in an office cafeteria
03

Reasoning takes longer still

A reasoning model picks the task and the target, then watches the execution. Only this layer runs off the robot.

So the robot is a stack: three layers, three clocks. The price: every layer needs an interface the layer above can write. Every work in this talk builds a layer or an interface.
Clips: SONIC project page (arXiv 2025, Science Robotics 2026) · GR00T 2 project page (NVIDIA whitepaper, Nov 2026) · numbers: GR00T 2 page; whitepaper Fig. 9, Table 27
StartRootsMoveHandsSystem 0SeeSystem 1
04
Foundations · the stack

Think slow, move fast

The 2026 stack on a Unitree G1. The rest of the talk builds it from the bottom up.
S2Reasoning modelpicks the task
and the target
S1GR00T 2 Turboworld-action model
imagines + plans
S0SONICwhole-body controller
balance and contact
Real Unitree G1 · table clean-up
WAM API
whole-body
tokens
joint
targets
proprioception · 50 Hz
camera images + proprioception
onboard · joint PD control
1 kHz
the robot’s own looptracks the joint targets
onboard · Jetson Orin NX
50 Hz
token in, joint targets outevery 20 ms
onboard · Jetson AGX Thor
≈ 1 Hz
each plan covers 1.6 s124 ms per prediction
runs off the robot
no rate published
optional: a VLM agent, ascripted plan, or an operator
Layer names follow the GR00T 2 whitepaper. In 2023, where this talk starts, neither green box existed: SONIC appeared in 2025, GR00T 2 Turbo in 2026.
Names and numbers: NVIDIA. GR00T 2 whitepaper, Nov. 2026 (Fig. 9, Table 27) and project page, “Think Slow, Move Fast” · clip: same page, speed not stated · tick strips are schematic
StartRootsMoveHandsSystem 0SeeSystem 1
05
Foundations · the lens

One stack, four questions

Read every work as one answer per row.
S2Reasoning model
S1GR00T 2 Turbo
S0SONIC
Unitree G1
WAM API
whole-body tokens
joint targets
training data
1
2
3
4
1S0 · controllerwhat moves the body?
two cooperating policiesRoboDuetRL legs + driven armsHOMIE · Decoupled WBCone specialist per skillHoST · BeamDojo · VB-Com · Gallantone tracker, any motionSONIC
2Commandwhat is sent down?
velocityRoboDuet · BeamDojogoal pointGallantcockpit joint anglesHOMIEcommand incrementsVIRALwhole-body tokensSONIC
3S1 · writerwho writes the command?
a personHOMIE · TAGRGB policy from simulationVIRAL · DoorMan · GRAILa VLAGR00T N1.5 on SONIC tokensa world-action modelGR00T 2 Turbo · onboard
4Datawhat trained it?
rewards in simulationHoST · GallantteleoperationHOMIE · TAG≈ 700 h motion captureSONICgenerated videoGRAILsimulated replaysGR00T 2 Turbo
The right end of every row is the 2026 stack. Each work in this talk moves one step along these rows.
Layer names: NVIDIA. GR00T 2 whitepaper, Nov. 2026 · 700 h: SONIC, Science Robotics 2026 (source data; 611 h after filtering) · each rung is detailed in its chapter
StartRootsMoveHandsSystem 0SeeSystem 1
06
Roadmap

Six questions, one thread

Each chapter answers one question; each work inherits a problem from the one before it.
specialists a person in the loop one controller eyes a model in the loop

1 · Roots

Before humanoids: can a robot touch, walk and work?
RoboDuet · pick-up (2×)
DiffTactile2024
RoboDuet2024
GRUtopia2024

2 · Move

Can a humanoid’s legs go anywhere, and get back up?
Gallant · under a ceiling
HoST2025
BeamDojo2025
VB-Com2025
Gallant2025

3 · Hands

Who drives the upper body, and where is the data?
HOMIE · the cockpit
HOMIE2025
FORGE2026
TAG2026
OPFA2026
also: TeleOpBench2025

4 · System 0

Can one controller execute any whole-body motion?
SONIC · boxing, real G1
Decoupled WBC2025
SONIC2025
also: SMAP2025

5 · See

Can a policy write the commands from the robot’s camera?
GRAIL · RGB pick-up
VIRAL2025
DoorMan2025
GRAIL2026

6 · System 1

Can one model write the commands for every task, onboard?
GR00T 2 Turbo · popcorn
GR00T 2 Turbo2026
Who writes the command: a persona policy trained in simulationa world-action model
StartRootsMoveHandsSystem 0SeeSystem 1
07
Chapter 1  ·  2023 – 2024

Roots

Before the humanoid: can a robot touch, walk and work?
DiffTactile ICLR 2024
RoboDuet RA-L 2025
GRUtopia Technical report 2024
08
ICLR 2024
Touch in simulation

DiffTactile: touch you can differentiate

A simulator of a soft tactile fingertip on rigid, soft and cable objects, differentiable from end to end.
Simulation · the gel fingertip follows a wavy surface
the same trajectory,
on the real sensor?
Real sensor · the same trajectory, replayed open loop (0.5×)
Simulation · two fingertips slide along a cable
a rule trained here,
on a real gripper?
Real gripper · a rule trained in simulation lifts a soft cube (2×)
Before · six tactile simulators
Rigid objects only, or no gradients
soft objects: 1 of 6differentiable: 1 of 6both: 0 of 6
both at once?
DiffTactile · one differentiable simulator
Rigid, soft and cable objects, with gradients
FEM gel sensorMLS-MPM / PBD objectsoptical simulation
DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation. ICLR 2024 · clips: project page · our count of Table 1
StartRootsMoveHandsSystem 0SeeSystem 1
09
How it works

Soft sensor, soft object, gradients through both

FEM gel sensorthe soft fingertip
Penalty contactforce + friction
MPM / PBD objectrigid · soft · cable
Lossobject state + touch
Trajectoryfingertip path
6 parametersgel stiffness · contact
markers · contact force
gradient: fit the simulator
gradient: tune the skill
Three models, one simulator. FEM for the soft gel, MLS-MPM or PBD for the object, a penalty contact model between them. All of it is written in Taichi, so every step can be differentiated.
Use 1: fit the simulator. Gradient descent tunes six physical parameters until the simulated markers match the real sensor: mean squared marker error 3.08 on real data, against 3.29 – 3.54 for the three baselines.
Use 2: tune the skill. The gradient also reshapes the fingertip’s trajectory; the loss can include touch: force, contact location, slip. In simulation, every method starts from the same trajectory and gets 100 iterations.
17.11°
lid openedbest baseline −0.68° (SAC)
60.82°
block turnedbest baseline 5.00° · goal 90°
Simulation: pushing a block up
what are the
gradients for?
Markers: yellow sim, green real
… and after
100 iterations?
Gradients: the lid opens
PPO baseline: lid stays shut

Unlocked

Touch you can simulate and differentiate: a contact-rich skill is tuned by gradients in 100 iterations.

Still open

Only fingertips on a fixed arm. What about a robot that must move its whole body to reach?

Clips: DiffTactile project page (2 s clips at half speed) · marker plot: paper Fig. 3 · numbers: Tables 2 and 5 of the paper (Table 5: simulation, best row per method) · the diagram is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
10
RA-L 2025
Legs + arm

RoboDuet: two policies, one body

One policy for the legs, one for the arm: together they track a 6D end-effector pose with the whole body.
Real Unitree Go1 + ARX5 arm: the trunk pitches down to reach the ground (2×)
and while
it walks?
Same robot on a glossy lobby floor: walking while the arm switches target poses
Before · one or the other
“Whole-body reach, or a full 6D pose?”
Deep WBC: position onlyGAMMA · GeFF: arm and legs separate
RoboDuet · both at once
“How must the legs lean to place the hand?”
6D end-effector posethe arm guides the trunkwhile walking
RoboDuet: Learning a Cooperative Policy for Whole-body Legged Loco-Manipulation. IEEE RA-L 10(5), 2025 · clips: project page, supplementary video
StartRootsMoveHandsSystem 0SeeSystem 1
11
How it works

The arm tells the legs how to lean

1 · Legs alone10,000 iterations
2 · Both together40,000 iterations
Base velocitywalk + turn speed
Loco policyMLP · 50 Hz
PD + robotGo1 + ARX5, 18 joints
Arm policyMLP · 50 Hz
6D hand poseend-effector target
12 joints
6 joints
state
state
guidance: body pitch + roll
PPO
PPO

Two commands, two policies. A base velocity drives the loco policy: 12 leg joints. A 6D end-effector pose drives the arm policy: 6 arm joints. Both run at 50 Hz, and PD controllers track their joint targets.

The arm policy has two extra outputs. A body pitch and a body roll: the guidance. They replace the loco policy’s own orientation command, so the arm can ask the legs to tilt the trunk.

Leaning enlarges the workspace. Targets on the floor or above the robot come within reach. In simulation 67.21 % of sampled poses are reached without a collision; a fixed-posture base with an IK arm reaches 52.38 %.

Legs first, then the duet. Stage 1 trains only the loco policy, arm frozen: a stable gait. Stage 2 trains both, sharing one end-effector tracking reward. Without stage 1, survival under 10–20 N pushes while standing falls from 98.2 % to 94.5 %.

Photo composite, real robot: walking right to left; each axis triad is one 6D target
does the trunk
follow the arm?
The trunk pitches with the arm
how far down
can it reach?
Down to the floor, in front of its legs
Clips: RoboDuet project page (locomanip-duet.github.io), real robot · numbers: Fig. 4 and Table III of the paper, simulation · composite: animated Fig. 1
StartRootsMoveHandsSystem 0SeeSystem 1
12
Under the hood

RoboDuet: Two PPO policies coupled by a pitch-and-roll guidance, trained in two stages

Deployment: the same two policies, no retraining
Target pose $\chi_t \in \mathbb{R}^{9}$spherical position $(l, p, y)$ + 6‑d rotation
Velocity command$v_x,\ v_y,\ \omega_{\text{yaw}}$ + trot clock of 4 sines
Arm policy $\pi_{\text{arm}}$MLP 512‑256‑128, ELU out: 6 joints + 2 guidance
Loco policy $\pi_{\text{loco}}$MLP 512‑256‑128, ELU out: 12 leg joints
PD + Isaac GymGo1 + ARX5 · 18 joint position targets · 50 Hz
Gait + velocity rewardtrot, 3 Hz steps, 0.06 m swing, energy penalty
PPOasymmetric actor-critic: the critic sees privileged simulation info
Pose reward $r_\chi$end-effector error vs target, paid to both policies
Operator, VR handlesright hand: velocity left hand: end‑effector pose
Both policies, frozen50 Hz, as in training
Real Go1 + ARX5, then Go2 + ARX5 zero-shotthe Go2 platform is 14.7 % heavier: 21.05 kg vs 18.35 kg
targets
state
targets
state
guidance: pitch, roll
state
reward
update
pose
reward
update
targets
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: velocity and pose in, 18 PD targets outStage 1: the legs learn to walk alone, with the arm frozenStage 2: the arm joins, and one pose reward pays both policiesDeployment: the same two policies on a real Go1, then a Go2What it costs, and what makes it work
Objective
  • Velocity: $r_{v} = \exp\{-|v_{x,y} - v^{\text{cmd}}_{x,y}|^{2} / \sigma_{v}\}$, weight $1.0$ in stage 1, $0.5$ in stage 2
  • Energy: $r^{\text{loco}}_{\text{energy}} = -0.00004 \sum_{i \in \text{legs}} |\tau_{i}\, \dot q_{i}|^{2}$, against motor overheating under the arm
  • Pose: $r_{\chi} = e^{-w\,\Delta(l,p,y)} \cdot e^{-\Delta(\alpha,\beta,\gamma)}$, $\ \Delta(\cdot) = \sum_{u} k_{u}\,|u_{t} - u^{\text{cmd}}|$, for both policies
  • Guidance: $r^{\text{arm}}_{\text{guide}} = -10\,|\phi_{t} - a^{\text{armG}}_{t}|$ ties the trunk’s pitch and roll $\phi_{t}$ to the guidance
Training recipe
  1. Stage 1, 10,000 iterations: train the loco policy alone; arm fixed at its default pose
  2. Stage 2, 40,000 iterations: add the arm policy; guidance replaces the pitch + roll command
  3. At the switch: footswing reward −10 → 0, hip constraint 0 → −0.05, velocity rewards halved
  4. Deploy both policies at 50 Hz; a person sends velocity and end-effector pose by VR handles
4,096parallel robots in Isaac Gym
50,000PPO iterations:
10,000 + 40,000
18joint targets at 50 Hz: 12 leg + 6 arm
3 seedsper method in the ablation
The detail that makes it work

Two guidance outputs of the arm policy replace the legs’ pitch and roll command. Workspace in simulation, standing still: 0.86 m³; one policy with two heads: 0.60 m³.

Versus its predecessor

vs Deep Whole-Body Control: two policies and a 6D pose instead of one policy and position only. vs Walk These Ways: its gait reward is the stage-1 prior.

StartRootsMoveHandsSystem 0SeeSystem 1
13
Results

What the duet buys on real robots

Doll from a 60 cm cabinet up to a 100 cm desk (2×)

Floor to desk, four heights

A doll goes from the ground to a chair, a cabinet and a standing desk, then back. The trunk tilts down to the floor, up to the desk.

… and targets outside
the training range?
Targets high above the robot: the trunk rolls and pitches

Extreme poses, few collisions

Five target poses, four of them extreme, eight tries each. Self-collision in 5 % of trials; 50 % with a fixed-posture base and IK.

… on a robot it was
never trained on?
Go1 (left), Go2 (right): same policies, same targets

Another dog, no retraining

The policies trained on a Go1 run unchanged on a Go2 that is 14.7 % heavier. A VR device sends both the same target poses.

39 / 60
RoboDuetreal transfers, 4 × 15 trials
32 / 60
Built-in controllersAFB+ID, of Go1 and ARX5
10 / 60
Fixed-posture base + IKFB+IK · 0 / 30 at the desk

Unlocked

Legs and arm act as one body: velocity and 6D hand pose in; the legs lean and step to reach it.

Still open

Only the low-level controller: blind, and a person supplies every target. What does the layer above need?

Clips: RoboDuet project page and supplementary video, real robots · numbers: Tables V and VI and Sec. V-B of the paper · 0 / 30: our sum of two columns of Table VI
StartRootsMoveHandsSystem 0SeeSystem 1
14
Technical report 2024
A simulated society

GRUtopia: a city to test the layer above

A simulated city where a language agent has to act through learned legged controllers.
Paper teaser · simulation renders: 100k scenes, a city, NPC residents
Simulation · a learned walking controller in the loop (Unitree G1, docs demo, 2025)
how did agents move before?
Before · most embodied-AI platforms
“Where should the agent go?”
motion by animationposition set directly
and with a body in the loop?
GRUtopia · learned controllers as APIs
“Can a walking robot get there?”
RL walking policiesfalls counted as resets89 scene categories
GRUtopia: Dream General Robots in a City at Scale. Technical report, arXiv:2407.10943, 2024 · teaser: paper Fig. 1 · clip: platform docs
StartRootsMoveHandsSystem 0SeeSystem 1
15
How it works

A slow agent on top of a fast controller

GRBenchtask + instruction
GRScenes100k scenes
GRResidentsLLM-driven NPCs
AgentLLM or VLM · 1 Hz
Path plannerRRT* · way-points
Walking policyRL · 60 Hz
Isaac Simphysics · 240 Hz
velocity
joint targets
joint state
goal
one RGB-D frame per second
dialogue

The fast loop. A walking policy trained with RL takes a velocity command and sets joint targets 60 times a second, on physics stepped at 240 Hz.

The slow loop. An agent built on an off-the-shelf LLM or VLM gets one rendered RGB-D frame per second and names a goal; an RRT* planner turns the goal into velocity commands.

A world with residents. GRScenes: about 100k scenes in 89 categories. GRResidents: NPCs driven by an LLM that reads the simulator’s ground truth. GRBench: three task families, 300 episodes each.

Two levels, one narrow interface. Language and pixels above, once a second; balance below, 60 times a second; between them a velocity command. It is the first two-level stack in this talk.

Physics240 steps Walking policy60 steps Agent1 decision one second
Schematic: one second of the loop, drawn to scale
what does
the agent see?
Paper fig. 7 · simulation: the H1, and its camera view
and whom
can it ask?
Paper fig. 5 · the agent asks an NPC which bowl is meant
Figures: GRUtopia paper, Figs. 5 and 7 (simulation) · rates: App. C.3 · velocity command: current repository · diagram and timeline are schematics
StartRootsMoveHandsSystem 0SeeSystem 1
16
Results

With a body in the loop, success drops

Simulation · flat ground (Unitree G1, docs demo, 2025)
100 %

Flat ground. Walk from point to point: the H1 and the Aliengo + Z1 both reach every goal.

… and between
furniture?
Paper fig. 7 · simulation: the H1 in a furnished home
58 % · 14 %

Furnished house, same controllers: H1 58 %, Aliengo + Z1 14 %. The paper blames mostly path and motion planning.

… and with a task
for the arm?
Paper fig. 5 · loco-manipulation: pick here, place there
0 %

Loco-manipulation in GRBench: no agent, with any LLM or VLM, completes a single episode.

Unlocked

The two-level split becomes measurable: a language agent has to act through a learned legged controller.

Still open

Under the agent is only a walker: 100 % on flat ground, 58 % in a furnished house. The legs come first.

Clip: platform docs (G1 demo of 2025, an illustration) · figures: paper Figs. 5 and 7 · numbers: Tables 3 and 4 of the paper (H1, Aliengo + Z1; simulation)
StartRootsMoveHandsSystem 0SeeSystem 1
17
Chapter 2  ·  2025

Move

Can a humanoid’s legs go anywhere,
and get back up?
HoST RSS 2025 · Outstanding Systems Paper finalist
BeamDojo RSS 2025
VB-Com ICRA 2026
Gallant CVPR 2026 · Highlight
18
RSS 2025 · Outstanding Systems Paper finalist

HoST: stand up
from almost anywhere

Real Unitree G1 · proprioception only
No motion data, single-stage RL
19
How it works

Four reward groups, four critics, a helping hand

Smoothness lossL2C2 · actor + critics
PPOadds 4 advantages
4 criticsone per reward group
4 reward groupsone weight per group
Pulling force200 N → 0 N
No commandno goal, no reference
Policy50 Hz · 23 joints
Action bound βlimits each joint step
PD controller500 Hz on the robot
Unitree G1Isaac Gym in training
action
target
torque
IMU + joint encoders · the last 5 states
state, reward
update
lifts the base

No command, no camera. The policy reads the IMU and joint encoders and moves 23 joints at 50 Hz. One behaviour: stand up, then stay standing. Trained with PPO in Isaac Gym, on four terrains.

One critic per reward group. Task, style, regularisation and post-task rewards each get a value network; PPO adds the four normalised advantages. With a single critic, success is 0.0 on all four terrains.

A helping hand that fades. Once the trunk is upright, 200 N pull the base up. Each time the head reaches its target height, the force drops by 20 N, down to 0. Without it: 0.0 success on ground, wall and slope.

Then calm it down for hardware. The action bound β shrinks from 1 to 0.25, and a smoothness loss keeps the outputs for neighbouring states close. Without the bound, the smoothness error on flat ground is 9.52, not 2.90.

Simulation: ground, platform, wall, slope
who scores
the attempt?
rightingrisingstanding task × 2.5 style × 1 regularisation × 0.1 post-task × 1 0.45 m0.65 m base height
Schematic: when each group pays
a fallen robot:
how does it explore?
200 N · β = 1 action bound β pulling force 0 N β = 0.25 1038 step a step: head at target height
Schematic: force and bound curricula
… and on
real hardware?
Real G1 · up from the lab floor
Clips: HoST code repository (simulation) · project page, archived copy (real robot) · numbers: Table III, Appendix B · both figures are schematics; steps 10 and 38 are our arithmetic
StartRootsMoveHandsSystem 0SeeSystem 1
20
Results

Twenty real trials, twenty stand-ups

Real G1 · the lab scene for the platform terrain

Four lab scenes, five trials each

An ottoman, a sofa back and a ramp stand in for platform, wall and slope. All 20 trials end standing.

… outdoors?
Real G1 · stone platform, tree, gravel path, lawn

Outdoors, on ground it never saw

Grass, a stone road, a wooden boardwalk: none of these terrains was in the training simulation.

… while it is
being kicked?
Real G1 · kicked while it is getting up

Kicked mid-rise, it still gets up

It does not go back to the floor. With a backpack: 3 of 3 trials at 10 kg, 2 of 3 at 12 kg.

20 of 20
real G1, 4 lab scenes5 of 5 in every scene
11 of 20
no smoothness losssame scenes · slope 0 of 5
≥ 94.2 %
simulation, 4 terrainssingle critic: 0.0 on all four

Unlocked

A humanoid no longer has to start on its feet: 20 of 20 real trials, with no motion data.

Still open

Blind, and it does one thing. On hard ground, where should the next step land?

Learning Humanoid Standing-up Control across Diverse Postures. RSS 2025 · clips: project page (archived copy) · numbers: Tables III, IV, V
StartRootsMoveHandsSystem 0SeeSystem 1
21
RSS 2025
Sparse footholds

BeamDojo: aim every footstep

A humanoid learns to put a flat foot on stepping stones and a 20 cm beam, reading an onboard LiDAR elevation map.
Real Unitree G1 · stepping stones, a platform, then beams, in one run
… and on a beam
20 cm wide?
Balancing beam, 20 cm wide: one foot in front of the other
Before · quadrupeds on sparse footholds
“Is the foot, a point, on a stone?”
feet modelled as pointsone misstep ends the episode
BeamDojo · a flat humanoid sole
“How much of the sole is on the stone?”
points sampled under each solea misstep first costs only a penalty
BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds. RSS 2025 · clips: project page (why618188.github.io/beamdojo)
StartRootsMoveHandsSystem 0SeeSystem 1
22
How it works

Practise on flat ground, pay for imagined missteps

PPOadds 2 advantages
2 criticsone per reward group
18 walking rewardsdense, every step
Foothold rewardsparse, per sole point
Velocity commandforward / backward
Policy50 Hz · 12 leg joints
PD controller500 Hz
Unitree G1Isaac Gym in training
Elevation map15 × 15 grid · 10 Hz
speed
targets
torque
LiDAR
height map + proprioception
update

A walker that reads a height map. A speed command, proprioception and a 15 × 15 grid of ground heights go in; targets for the 12 leg joints come out at 50 Hz. On the robot, a LiDAR builds the grid at 10 Hz.

A reward for a flat foot. A quadruped’s foot is a point; a humanoid’s sole is an area. So sample points under each sole: every point that hangs over a hole costs a penalty, and more overlap with the stone pays more.

Two critics, one per kind of reward. One learns the 18 dense walking rewards, the other only the sparse foothold penalty; PPO adds their normalised advantages. With one critic, success on hard stones is 55.67 %, not 91.67 %.

Practise on flat ground first. Stage 1: the feet touch flat ground while the map shows the stones, so a misstep costs a penalty and the episode goes on. Stage 2: real stones, where a misstep ends it. Without stage 1: 42.00 %.

Simulation · red dots = height-map samples
what is a good
footstep?
over a hole:one penalty each on the stone:no penalty stonehole
Schematic: points under one sole
a sparse reward:
who learns it?
fromto two critics one critic:corrects late
Schematic, after Fig. 7: foot paths
one misstep ends
the episode?
Stage 1 · flat ground penalty blue = holes in the map misstep = penalty, go on Stage 2 · real stones episode ends misstep = episode over
Schematic: stage 1 and stage 2
Clip: BeamDojo project page (simulation) · numbers: Table II of the paper, hard stepping stones in simulation · 18 rewards = our count of Table VII · the three figures are schematics
StartRootsMoveHandsSystem 0SeeSystem 1
23
Results

Zero-shot onto stones and a 20 cm beam

Real G1 · stepping stones, 20 cm in size

No fine-tuning on the robot

Stones up to 45 cm apart. In training, the height map gets four kinds of noise.

… on terrain it
never trained on?
Real G1 · a 50 cm gap, not a training terrain

Terrains it never trained on

Gaps: 5 of 5 on the real robot. Stepping beams: 3 of 5. Neither was a training terrain.

… walking
backward?
Real G1 · the same stones, walking backward

Backward over the same stones

The LiDAR map also covers the ground behind. One depth camera cannot handle this.

4 of 5
stepping stones, real G1without map noise: 1 of 5
4 of 5
20 cm beam, real G1without map noise: 0 of 5
91.67 %
simulation, hard stonesPIM baseline: 46.67 %

Unlocked

A flat foot lands on stepping stones and a 20 cm beam, straight from an onboard LiDAR elevation map.

Still open

It can only step where the map says; map drift still limits it. What happens when the map is wrong?

Clips: BeamDojo project page (why618188.github.io/beamdojo) · numbers: Fig. 8 (real robot, 5 trials per terrain) and Table II of the paper
StartRootsMoveHandsSystem 0SeeSystem 1
24
ICRA 2026
Unreliable perception

VB-Com: when the eyes fail, fall back on reflexes

A vision policy and a blind policy run side by side; the robot switches to the blind one when perception fails.
Real Unitree G1, head lidar working: it steps over both hurdles
two hurdles again,
with the head sensor covered?
Real G1, head sensor covered: it walks into both hurdles and recovers
Before · a vision policy alone
“Where does the map say I can step?”
needs a correct height mapneeds static terrain
and what does
VB-Com ask?
VB-Com · adds a second question
“Is this still going well?”
answered from proprioception alonea blind policy runs alongside
VB-Com: Learning Vision-Blind Composite Humanoid Locomotion Against Deficient Perception. ICRA 2026 · clips: project page
StartRootsMoveHandsSystem 0SeeSystem 1
25
How it works

Two policies, and an estimate of which does better

Height mapfrom the head lidar
Proprioceptionjoints, base, command
Vision policy $\pi_v$map + proprioception
Blind policy $\pi_b$proprioception only
Return estimatorsone per policy
Switcha rule, not a network
Unitree G1 · H1runs one action
action $a_v$
action $a_b$
joint and base state, every control step
history
estimated returns $\hat G_v,\ \hat G_b$
action $a$

Two policies, trained the same way. Same rewards, same action space, PPO in simulation. The vision policy reads a height map from the head lidar; the blind policy reads proprioception only.

Each policy gets a return estimator. A small network predicts how much reward its policy will collect over the next switch period. Its only input is a history of proprioception.

A fixed rule picks one action. The vision action runs while its estimate is above the blind one, and the blind estimate has not fallen below its 5-step mean minus a margin. Otherwise the blind action runs.

A wrong map cannot fool the switch. The estimators never see the map. A collision or a missed step shows up in proprioception: both estimates drop within several control steps, and the blind policy takes over.

The drop detector carries the result. Without the threshold on the blind estimate, goals completed fall from 85.76 % to 48.48 % in simulation. The switch stays a rule with hand-set parameters.

Real G1: around a standing person
how does a policy know
it is doing well?
vision estimate blind estimate threshold vision action runs collision both drop in a few steps blind action vision again
Schematic after the paper’s Fig. 5: two return estimates
and when the map
is wrong?
Real H1: a missed step (0.5×)
does the threshold
matter?
85.76 %with the threshold
48.48 %without it
goals completed · simulation
Ablation of the switch threshold
Clips: VB-Com project page (renjunli99.github.io/vbcom.github.io) · switch rule: Eqs. 7, 8 · numbers: Table IV, simulation · the curve figure is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
26
Results

Corrupt the map, and success barely moves

Real Unitree G1: a person runs into its path

Hit, switch, walk around

The map misses a person who runs in. The G1 is hit, switches to the blind policy and gets around him.

and a hurdle that appears
in mid-stride?
Real Unitree H1: two beams dropped in its path

Hurdles that appear too late

Beams fall into the H1's path as it walks. It collides, recovers on the blind policy, then does it again.

how often does it
reach the goal?
goals completed 80 %60 %40 % 03070100 % perception noise VB-Com 84.81 % Blind 83.76 % Vision 48.71 % 73.57 %
Simulation · goals completed vs perception noise

Noise barely moves it

Goals completed stay at 82–85 % from 0 to 100 % noise. The vision policy alone falls from 73.57 % to 48.71 %.

84.81 %
VB-Comgoals completed · 100 % noise
80.52 %
Noisy Perceptive policytrained on the test noise
1.50 %
VB-Com collision steps0 % noise · blind policy: 2.57 %

Unlocked

The robot no longer has to trust its eyes: a blind reflex policy plus a learned “is this going well?” signal.

Still open

Two policies, a hand-set switch, a map that cannot show what is overhead. Can one policy see the full 3D space?

Clips: VB-Com project page (renjunli99.github.io/vbcom.github.io), real robots · numbers: Table II of the paper, simulation, hardest terrain level
StartRootsMoveHandsSystem 0SeeSystem 1
27
CVPR 2026 · Highlight

Gallant: one voxel grid
for floor, walls and ceiling

One policy · a real Unitree G1
The whole course, one take
28
The idea

A height map has no ceiling

A 2.5D map keeps one height per ground cell. A voxel grid keeps the whole space around the robot.
Real Unitree G1: around two panels, then under a hanging one
what does the policy
get to see of this scene?
What the policy gets in a door-and-ceiling scene: 5 cm voxels
Before · height map (2.5D)
“How high is the ground in this cell?”
? floor or panel? schematic
one number per ground cellreal low ceiling: 0 of 15real door: 1 of 15
Gallant · voxel grid
“Is this 5 cm cell of space occupied?”
occupied free schematic
32 × 32 × 40 cells around the torsoreal low ceiling: 15 of 15real door: 15 of 15
Gallant: Voxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains. CVPR 2026 · clips: project page · numbers: Fig. 6
StartRootsMoveHandsSystem 0SeeSystem 1
29
How it works

Height slices as channels, one policy end to end

LiDAR simulationscans its own links
Goal positionrelative to the robot
PPOgoal-reaching reward
Two torso LiDARsfront + back · 10 Hz
Voxel grid32 × 32 × 40 · 5 cm
2D CNNheight = channel axis
Policyone MLP actor · 50 Hz
PD + Unitree G129 joint targets
points
10 Hz
64-d
25 Hz
50 Hz
proprioception
in training
update

Two LiDARs, one grid. Two Hesai JT128 on the torso, front and back, cover ∼4.00π of solid angle; elevation-map methods see ∼1.97π.
Ten times a second, their points become a binary grid of 5 cm cells.

Height becomes the channel axis. Most columns hold one or two occupied cells, so a plain 2D CNN reads the 40 height slices as 40 channels. A training iteration takes 5.5 s; with a 3D CNN, 10.3 s.

A goal point in, 29 joint targets out. The command is a position relative to the robot, not a velocity. One policy does local navigation and locomotion, at 50 Hz, on the onboard NVIDIA Orin NX.

One stage of PPO, all in simulation. Eight terrain families with a curriculum; the reward pays for reaching the goal within 10 s.
The simulated LiDAR also returns the robot's own moving links.

Randomize the LiDAR, or it fails on hardware. Pose, hit points, 100–200 ms of latency, 2 % of voxels masked. Without it, real successes in 15 trials fall: platform 14 → 4, stairs up 15 → 6.

Voxel grid while climbing stairs
how does a network
read 3D cheaply?
40 height slices = 40 channels z +1.0 m −1.0 m 2D CNN → 64-d
Schematic: height slices as channels
and what is
the command?
goal point robot Pt its own path
Schematic: the command is a point
where is it trained,
and does it transfer?
Isaac Sim training terrain: crouching under ceiling slabs
Gallantno LiDAR randomization1515Ceiling1215Door414Platform912Stones615Stairsup814Stairsdown
Real robot: successes in 15 trials per terrain
Clips: project page (gallantloco.github.io) · numbers: Table 1, Fig. 5d, Sec. 3.2 and Fig. 6 of the paper · the two middle figures are schematics
StartRootsMoveHandsSystem 0SeeSystem 1
30
How it trains

Simulate the sensor, not only the terrain

Scene meshesterrain + robot links
LiDAR ray-castwith NVIDIA Warp
Randomizationpose, hits, lag, mask
Two real LiDARsHesai JT128 · 10 Hz
OctoMaponboard · Orin NX
Voxel grid32 × 32 × 40 · 10 Hz
Policyone policy · 50 Hz
points
on the robot
10 Hz
meshes
points
in training

All the policy sees of the world is this grid. On the robot, two LiDARs and OctoMap build it ten times a second.
In training, a simulator has to build the same grid, with the same flaws.

Cast rays at everything, the robot included. Each mesh, terrain or robot link, keeps a precomputed BVH in its own frame; the rays are moved into that frame, so nothing is rebuilt. One NVIDIA Warp kernel covers all environments.

Leave the body out, and success collapses. Simulation only, hardest setting: low ceiling 97.1 → 28.4 %, platform 96.1 → 22.16 %,
stairs up 96.2 → 33.0 %. Standing upright it hardly matters: door 98.7 → 98.3 %.

Then make the scan late and noisy. LiDAR pose: 1 cm and 1° of noise per episode. Every hit point: 1 cm.
Each scan arrives 100–200 ms late, and 2 % of the voxels are masked.

Leave that out, and the robot reacts too late. Real robot only, successes in 15 trials: platform 14 → 4, stairs up 15 → 6,
stairs down 14 → 8, stepping stones 12 → 9. Under the low ceiling nothing changes: 15 → 15.

Voxel grid: a platform and a gap
what does a real
LiDAR also hit?
Paper Fig. 5 · with self-scan, without
and if the body
is left out?
Simulation · success rate, %self-scanwithout itCeiling97.128.4Platform96.122.16Stones82.127.2Stairs up96.233.0Door98.798.3
Simulation only · Table 3a, redrawn
what else is different
about a real scan?
LiDAR pose1 cm, 1° · per episodeHit points1 cm · on every hitLatency100–200 ms latescanseenVoxels2 % masked
Schematic: the four LiDAR randomizations
Real robot, no LiDAR randomization · paper Fig. 9
Clip: project page · stills: Fig. 5 and Fig. 9 of the paper · numbers: Sec. 3.2, Table 3a (simulation only), Fig. 6 (real robot only) · the chart is redrawn, the icon figure is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
31
Under the hood

Gallant: Simulated LiDAR with self-scan, a z-as-channel CNN, one-stage PPO

Privileged inputsbase velocity $v_t$ height map 33 × 33 = 1,089
Critic $V$same structure as the actor, separate weights
PPOclip 0.2 · $\gamma$ 0.99 · $\lambda$ 0.95 lr 5e−4 · entropy 0.003
Reward $r_t$goal reaching + direction, head height, foot clearance
Voxel grid $X_t$binary 40 × 32 × 32 · 5 cm ±0.8, ±0.8, ±1.0 m · 10 Hz
z-grouped 2D CNN3 conv: C = 8, k = 3, s = 2 Mish · linear → 64-d
Actor $\pi(a_t \mid o_t)$MLP 256 + CNN 64 → 256 → 29 joint targets · 50 Hz
PD + Isaac Sim29-DoF Unitree G1 1024 × 8 environments
Real Unitree G1Orin NX · OctoMap 10 Hz FastLIO2 goal 25 Hz
LiDAR ray-cast (Warp)2 × JT128: terrain and the robot's own moving links
LiDAR randomizationpose, hits: $\sigma$ = 1 cm, 1° lag 100–200 ms · 2 % masked
Goal + proprioceptiongoal $P_t$, time elapsed / left history: 6 states, 4 actions
Terrain curriculum8 families · difficulty $s$ 8 m blocks · 10 s episodes
64-d
PD targets
proprioception
points
noise
GAE($\lambda$)
value loss
update
terrain
zero-shot
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: voxel grid and goal in, 29 joint targets outThe simulated LiDAR sees the robot's own body, late and noisyOne-stage PPO; only the critic sees velocity and the height mapA terrain curriculum in training, then zero-shot on the robotWhat it costs, and why the self-scan matters
Objective
  • 2D conv over $x$–$y$: $Y_{o} = \sigma\big(\sum_{c=0}^{C-1} W_{o,c} * X_{c} + b_{o}\big)$, $\ C = 40$ height slices as input channels
  • LiDAR: $P^{\text{rand}} = P + \mathcal{N}(0, 1)$ cm for pose and hit points, rays $+\ \mathcal{N}(0, (\pi/180)^{2})$ rad
  • $r_{\text{reach}} = \mathbb{1}(t > T - T_r)\,/\,\big[T_r\,(1 + \|P_t\|^{2})\big]$, $\ T = 10$ s, $T_r = 2$ s; $\ r_{\text{head}} = \exp\big(-4\,(H^{\text{est}}_{\text{head}} - H_{\text{head}})^{2}\big)$
  • Curriculum: $p_{\tau}(s) = (1 - s)\,p^{\min}_{\tau} + s\,p^{\max}_{\tau}$, e.g. ceiling 1.30 → 1.00 m, stair step 0 → 0.20 m
Training recipe
  1. Ray-cast both torso LiDARs with Warp against the terrain and the robot's own links
  2. Randomize LiDAR pose, hit points and latency (100–200 ms); mask 2 % of the voxels
  3. Run PPO for 4,000 iterations; the critic also gets base velocity and a height map
  4. Promote to harder terrain on success, demote on failure; deploy without fine-tuning
1024 × 8environments on eight RTX 4090 GPUs
4,000PPO iterations of 5.5 s each (3D CNN: 10.3 s)
8terrain families in one curriculum
50 Hzpolicy · voxel grid 10 Hz, goal position 25 Hz
The detail that makes it work

The simulated LiDAR also scans the robot's own links. Without that, simulated success falls from 97.1 % to 28.4 % under the ceiling and from 96.1 % to 22.16 % on platforms.

Versus its predecessor

vs BeamDojo, VB-Com: a voxel grid replaces the elevation map (field of view ∼4.00π vs ∼1.97π). vs HOMIE: the same reward family, with goal reaching in place of velocity tracking.

StartRootsMoveHandsSystem 0SeeSystem 1
32
Low ceiling · 15 of 15
Door · 15 of 15
and over obstacles
on the ground?
Stairs · up 15, down 14 of 15 · here right to left
Left to right · 15 cm steps left, 20 cm right
and with a gap
to cross?
Platform with a gap · 14 of 15 · here the narrowest gap
The same 30 cm platform · the widest of the three gaps
Gallant · real robot

Under and around. A low ceiling and a door: 15 of 15 each, straight from simulation.

Stairs, both ways. Up one flight and down the other: 15 of 15 up, 14 of 15 down.

A 30 cm platform with a gap. 14 of 15. The project page shows gaps of 30, 35 and 40 cm.

Seven terrains, fifteen trials each, one policy

15 of 15Flat floor
15 of 15Low ceiling
15 of 15Door
14 of 15Platform + gap
12 of 15Stepping stones
15 of 15Stairs up
14 of 15Stairs down
100 of 105our sum of Fig. 6
StartRootsMoveHandsSystem 0SeeSystem 1
33
Stepping stones (0.5×) · 12 of 15 · height map 10
12 of 15on the real robotthe lowest of seven terrains
82.1 %in simulationhardest setting · Table 3
> 90 %zero LiDAR latencyin simulation · no table printed
Stepping stones: real robot and simulation
and a height-map policy,
on the same robot?
Gallantheight-map policy1515Flat015Ceiling115Door1114Platform1012Stones1315Stairsup1214Stairsdown
Real robot: successes in 15 trials per terrain
Gallant · results

The weak spot. Stepping stones, 12 of 15: every LiDAR scan is over 100 ms old.

What a height map cannot show. Low ceiling 0 of 15, door 1 of 15; with the voxel grid, 15 of 15 each.

One policy: over, around and under

Unlocked

One onboard policy for the full 3D space: it walks over, around and under obstacles, zero-shot from simulation.

Still open

It only walks to a goal point: no hands, no task.
Who drives the upper body?

StartRootsMoveHandsSystem 0SeeSystem 1
34
Chapter 3  ·  2025 – 2026

Hands

Who drives the upper body,
and where do demonstrations come from?
HOMIE RSS 2025
FORGE arXiv 2026
TAG arXiv 2026
OPFA ICRA 2026
35
RSS 2025

HOMIE: a $500 cockpit
for a whole humanoid

One seated operator · a real Unitree G1
Legs by pedal, arms by exoskeleton
36
How it works

Pedals drive learned legs, an exoskeleton sets the arms

Exoskeleton + gloveson the operator’s arms
Foot pedalunder the operator’s feet
Joint angles $q_{\text{upper}}$2 × 7 arm joints + fingers
Command $C_t$speed · turn · torso height
Policy $\pi_{\text{loco}}$RL · 50 Hz on the robot
PD + Unitree G1real robot · Dex3-1 hands
leg targets
copied joint for joint · nothing learned
state of every joint, arms included

One operator, two channels. A seated person wears two exoskeleton arms with gloves and rests both feet on a pedal board. The robot is split the same way: its arms follow the hands, its legs follow the feet.

The feet give three numbers. Forward speed, turning speed and torso height. A policy trained by reinforcement learning turns them into leg joint targets at 50 Hz, on the robot: it walks, and it squats.

The arms are not learned. The exoskeleton has the robot arm’s joint layout, so each robot joint copies one exoskeleton joint; gloves with Hall sensors set the fingers. No inverse kinematics, no pose estimation.

The legs watch the arms. The policy observes every joint of the robot, arms included, and it is trained while the arms move at random. That is how the legs stay balanced while the operator moves the arms freely.

It holds in the test. 1,000 simulated G1s, 20 s each, with fully random arm poses: on average they stay up 19.947 s of 20 s, with a mean torso-height error of 0.022 m.

Cockpit: exoskeleton, gloves, pedal
what do
the feet say?
Real G1: squats while the arms move
and the arms?
Real G1: arms and fingers in motion
moving arms shift the balance:
who tells the legs?
Isaac Gym: G1s train with random arm poses
HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit. RSS 2025 · clips: project page · numbers: Table II, App. C-A
StartRootsMoveHandsSystem 0SeeSystem 1
37
How it is built

No motion capture, no GPU, $500

CAD render: exoskeleton arms and gloves on a back plate
CAD render: three pedals, two mode buttons
Joint matchingno motion capture
1 : 1one sensor per joint
Arms2 × 7 joints · servo encoders, 0.09° Gloves2 × 15 DoF · Hall sensors Pedal3 pedals · rotary potentiometers
Command rateno GPU
263 Hzarms · hands 293 Hz
HOMIE263 Hz · no GPU AnyTeleop125 Hz · RTX 3090 OpenTelevision60 Hz · M2 chip Telekinesis16 Hz · 2 × RTX 3080 Ti
Costbill of materials
$500parts sum to $510
exoskeleton $430 gloves $30 each pedal $20 the paper’s VR baseline, OpenTelevision: $4k (Table I)
The linkWi-Fi to the robot
16 mslatency · 128-byte packets
cockpit263 Hz arm commands G1 arms10 Hz, interpolated on the robot
Arms, gloves, a pedal board. The exoskeleton arms are 3D-printed, with one small servo in every joint; the gloves read the fingers; three pedals and two mode buttons sit under the feet. No camera has to track the operator.
No motion capture. The servo in each joint reads its angle, as an absolute encoder accurate to 0.09°. The exoskeleton has the robot arm’s joint layout, so that angle is the robot’s joint target: no pose estimation, no inverse kinematics.
No GPU. Copying angles costs almost nothing: the cockpit outputs 263 arm and 293 hand commands per second without a GPU. The vision-based interfaces in the paper’s Table IV reach 16 to 125 Hz for the arms, on a GPU or an M2 chip.
About $500 of parts. $430 for the exoskeleton, $30 per glove, $20 for the pedal. It is not the cheapest system in the paper’s Table I, but the only row that ticks loco-manipulation, whole-body control and no MoCap together.
What reaches the robot. A packet is 128 bytes; the measured Wi-Fi latency is 16 ms. On the G1 the arm targets are updated at 10 Hz and interpolated on the robot; the paper states that the cockpit could support over 200 Hz.
Clips: CAD renders from the HOMIE project page (homietele.github.io) · numbers: Tables I, III and IV, Sec. III-C, Sec. IV-C1 and App. C-A of the paper
StartRootsMoveHandsSystem 0SeeSystem 1
38
Under the hood

HOMIE: Leg policy trained under random arm poses, with a knee reward and mirrored data

Critic $V$sees the true base velocity
PPOasymmetric actor–critic
Reward $r$ · 33 termstrack speed, turn, height · squat knee · regularizers
Command $C_t$$[v_x,\ \omega_{\text{yaw}},\ h]$ · resampled every 4 s
Policy $\pi_{\text{loco}}$6-step history → lower-body joint targets · 50 Hz
PD + Isaac GymUnitree G1 · 4,096 envs flat ground
Random arm targets$a_i \sim U(0, \rho'_a)$ · resampled every 1 s
Curriculum $\rho_a$0 → 1 in steps of 0.05
Symmetry losses$\text{MSE}(a_t, a'_t)$ · $\text{MSE}(V_t, V'_t)$
Mirror left ↔ rightabout the x–z plane · both copies are stored
Deployment: cockpitexoskeleton sets the arm joints, the pedal sets $C_t$
PD targets
state: every joint
state
reward
update
value
arm joints
range
velocity tracking good: widen the range
replaces
pairs
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The loop: three commands in, leg joint targets out, state backPPO on a 33-term reward (our count): track commands, bend kneesArm targets are random; their range grows with the legs’ skillEvery transition is stored twice: as it is, and mirroredWhat it costs, and what makes it work
Objective
  • Tracking: $\exp\!\big(-4\,\|v_x - v_{r,x}\|^2\big)$, weight 1.5; $\ \exp\!\big(-4\,\|h_t - h_{r,t}\|\big)$, weight 2.0
  • Squat knee: $r_{\text{knee}} = -\big\|(h_{r,t} - h_t)\,(\tilde q_{\text{knee},t} - \tfrac{1}{2})\big\|$, $\ \tilde q_{\text{knee}}$ = knee angle scaled to $[0, 1]$ over its range
  • Arms: $a_i \sim U(0, \rho'_a)$, $\ p(\rho'_a \mid \rho_a) \propto e^{-20(1-\rho_a)\rho'_a}$ on $[0, 1]$; uniform as $\rho_a \to 1$
  • $L^{\text{actor}}_{\text{sym}} = \mathrm{MSE}(a_t, a'_t)$, $\ L^{\text{critic}}_{\text{sym}} = \mathrm{MSE}(V_t, V'_t)$; primes = the mirrored copy
Training recipe
  1. PPO in Isaac Gym, 4,096 flat-ground environments; new commands every 4 s, one third squat
  2. Randomize payloads (torso −5 to 10 kg, hands −0.1 to 0.3 kg), friction, gains; push every 4 s
  3. Give the arms new random targets every 1 s; raise $\rho_a$ by 0.05 when velocity tracking is good
  4. Store every transition and its mirror image; add the two symmetry losses to the PPO loss
≈ 3 hon one RTX 4090 (project page)
4,096parallel environments
0 hof motion capture
50 Hzon the G1’s Jetson Orin
The detail that makes it work

Mirrored data: without it the velocity error is still ≈ 0.6–0.7 m/s after 2,000 steps, with it ≈ 0.19 m/s. The arm curriculum keeps balance under moving arms.

Versus its predecessor

vs HumanPlus, OmniH2O, Mobile-TeleVision: no retargeted MoCap prior, and torso height is a command. vs upper-body-only teleoperation: the legs are in the loop.

StartRootsMoveHandsSystem 0SeeSystem 1
39
Squat to a low shelf, place the tape roll higher (2×)
Squat to the ground and pick up a bottle (2×)
and when the task
needs walking?
Push a seated 60 kg person across a lobby
Carry a box from one shelf rack to the next (2×)
and with two hands,
or two robots?
Two G1s, two operators: an apple changes robots
One G1: a tomato passes from hand to hand (2×)
HOMIE · real robot, teleoperated

From the floor to the shelf. The G1 squats for a tape roll on a low shelf, and for a bottle on the ground.

Walking is part of the task. A 36.4 kg G1 pushes a seated 60 kg person, and carries a box to the next rack.

Two hands, or two robots. A tomato changes hands; with two operators, an apple passes between two G1s.

One operator, the whole body

One seated operator per robot, linked by Wi-Fi: pedals for the legs, exoskeleton and gloves for arms and fingers.
One seated operator per robot, linked by Wi-Fi: pedals for the legs, exoskeleton and gloves for arms and fingers.
Demonstrations, not a benchmark: one recorded run each; the paper reports no success rate or attempt count for them.
StartRootsMoveHandsSystem 0SeeSystem 1
40
Why it matters

From teleoperation to autonomy

Operatorin the cockpit
Commands $C_t$, $q_{\text{upper}}$3 numbers · joint targets
RL legs + PD + G1$\pi_{\text{loco}}$ unchanged · 50 Hz
Recorder · 10 Hz3 views · state · commands
Dataset2 tasks × 50 episodes
Policy $\pi_{\text{auto}}$Seer transformer · 10 Hz
first-person camera view
imitation
3 views + joint state

A person writes two things. Three pedal numbers $C_t$ and the upper-body joint targets $q_{\text{upper}}$. Everything below them, the learned legs and the PD loop, already runs on the robot.

Record what the robot sees and is told. Two tasks, 50 episodes each, at 10 Hz: three RGB views (head and both arms), the joint state, and both commands.

A policy takes the operator’s seat. Seer, an autoregressive transformer, reads the three views and the joint state; three heads output arm joints, hand joints and the height command. The legs are untouched.

It succeeds in most trials. 15 per task on the real G1: Squat Pick 73.3 %, Pick & Place 80.0 %. There is no baseline, and the policy runs off the robot: 10 Hz on an RTX 4080 host, over a cable.

The same cockpit drives simulated robots. The leg policies of the G1 and the Fourier GR-1 also run in Isaac Sim scenes from GRUtopia. Teleoperation only: no success rate, and no simulated data in training.

Cockpit: the G1 copies the operator, toy from hand to hand
and without
the operator?
Autonomous: Squat Pick
how often
does it work?
Autonomous: Pick & Place
and in
simulation?
Simulation: a G1 in a GRUtopia scene
Clips: HOMIE project page · numbers: Sec. IV-D, Table V, Sec. IV-C3, App. C-B and C-C of the paper · the diagram is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
41
Results

Faster than VR, quick to learn, and reused

Open Oven: one of the four timed desk tasks

Nearly half the time of VR

Four desk tasks against the VR system OpenTelevision: nearly half the time, says the paper; 1.25× to 2.05× faster per task.

and for someone
who has never tried?
Hand Over: the task of the user study

Five novices, five tries

People who had never teleoperated a humanoid, after a brief tutorial: on average 24.8 s on the first try, 8.0 s on the fifth.

and after
the paper?
HOMIE’s leg controllerRSS 2025 VIRAL · DoorManCVPR 2026 learned policies write its commands SONICScience Robotics 2026 its specialist baseline, “OpenHomie” Decoupled WBCper OpenHomie README “homie v2 rl policy”, in NVIDIA’s repository
Schematic: three later uses of the leg controller

What later works did with it

Learned policies write its commands where a person did; SONIC measures itself against it; NVIDIA’s repository holds a second version.

up to 2×
faster than VRvs OpenTelevision · 4 tasks
8.0 s
novices, fifth try24.8 s on the first · mean of 5 people
Feb 2025
code public sinceRL training + G1 deployment

Unlocked

One person drives a whole humanoid: walking, squatting, both hands at once. Every minute can be training data.

Still open

The body is split in two, and it feels no force. Can legs and arms answer to one controller that feels contact?

Clips: HOMIE project page · numbers: Sec. IV-C1, Fig. 11 (read off the bars), Table X · later works: their papers and the OpenHomie README · the figure is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
42
arXiv 2026
Force-aware whole-body control

FORGE: one policy for legs and arms that feels force

It takes velocity, foothold, wrist-pose and force commands, and estimates wrist force with no force sensor.
Walk and carry · a foam cubePaper figure · real robot
Reach a target · wiping a whiteboardPaper figure · real robot
Hold a contact · peeling a cucumberPaper figure · real robot
Sense a load · bottles into a basketPaper figure · real robot
+ +
Four command groupsbase velocity
footholds
wrist poses
wrist force (optional)
G1 humanoid29 joints
no wrist force sensor
Frozen leg experttrained first, to walk
Residual policylegs, waist and arms
Momentum observerarm model, not learned
Learned correctionfrom simulated contact
one whole-body policy
unused groups masked
29 joint targets · 50 Hz
wrist-force estimate
arm joint states
shifts the wrist target

Four kinds of command. A base velocity, footholds for both feet, a pose for each wrist and, optionally, a force each wrist should apply. Any subset can be active, and no human motion data is used in training.

One policy, on a frozen leg expert. A walking policy is trained first and frozen. The new policy adds small corrections for the legs and all of the arm motion. Command groups that are switched off are masked out.

Force, without a force sensor. A model-based momentum observer on the arm joints, plus a learned correction, estimates the force at each wrist. The force error then shifts the wrist target (gain Kf = 200 N/m).

FORGE: Force-Aware Multi-Granularity Control for Efficient Humanoid Loco-Manipulation. arXiv:2611.17142, 2026 · stills: Fig. 1 of the paper
StartRootsMoveHandsSystem 0SeeSystem 1
43
Results

A gentler grip, with no force sensor

Paper figure · real robot: force loop on (left), off (right)

The force loop softens the grip

Same policy, same lift, and both succeed. With the loop on, the foam cube is visibly less deformed. A photo: no number is given.

and how good
is the estimate?
Paper figures · real robot: orange = estimate, black = gauge

The estimate, against a gauge

A handheld gauge loads the wrist up to 20 N. RMSE on x / y / z: 0.81 / 1.97 / 0.87 N; the observer alone: 2.83 / 5.15 / 3.02 N.

and when a policy
writes the commands?
Head camera + task state Diffusion Policy 320 demonstrations FORGE 20 of 2020 of 2018 of 2018 of 20 empty box0.5 kg1 kg2 kg
Schematic: one dot per trial, real robot (Table V)

A policy can write the commands

A Diffusion Policy runs a squat-and-lift on the real robot: 20, 20, 18 and 18 of 20, from an empty box to 2 kg. No baseline is reported.

8.52 N
squeezing forcesimulation · rigid box test
15.48 N
HOMIEthe same simulated test
15.32 N
SONICsame test · TWIST2: 17.16 N

Unlocked

One controller for legs and arms: it estimates the force at each wrist without a force sensor.

Still open

The robot now senses the force; the operator still feels nothing. Can the person feel it too?

Stills: Fig. 8a, 9a and 9b of the paper · numbers: Tables I, IV and V · Table IV covers the traces of Fig. 8, not all trials · the dot figure is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
44
arXiv 2026
Tactile teleoperation

TAG: let the operator feel what the robot feels

A glove that tracks 21 finger joint angles and raises the robot’s fingertip contact on the operator’s skin.
Tracking only: an XHand on a stand mirrors every finger of the gloved hand
and when the fingers
touch something?
Pinching a 1.75 mm filament: fingertip sensor maps (top), bumps on the glove (inset)
The problem: the operator sees the robot hand but feels nothing. Did the finger touch? How hard is it pressing? Only the eyes can tell.
TAG returns the touch: 32 bumps on every fingertip show where the robot is touched and how hard. 21 joints are tracked to under 1°, and the open-source glove costs under $500.
Feel Robot Feels: Tactile Feedback Array Glove for Dexterous Manipulation. arXiv:2603.28542, 2026 (in submission) · clips: project page
StartRootsMoveHandsSystem 0SeeSystem 1
45
How it works

A magnet at every joint, 32 bumps per fingertip

Operator’s handin the TAG glove
21 joint encodersmagnet + 3-axis sensor
Hand retargetingfingertip poses → joints
Robot handtactile sensors
on its fingertips
Tactile mappingshape mode · pressure mode
32-bump arrayon every fingertip · 2 cm²
motion
21 angles · 260 Hz
joint commands
tactile map
32 × up / down
touch

A magnet at every joint. Each of the glove’s 21 joints carries a ring magnet over a 3-axis magnetometer. The angle is the atan2 of two field components: a ratio, so common-mode errors such as temperature cancel.

Touch comes back to the fingertip. The robot’s fingertip tactile map switches 32 bumps on the operator’s fingertip: electro-osmotic pumps, where 200 V across a membrane moves liquid and raises a 1.6 mm-wide dot.

Shape mode keeps the outline. The tactile image is resampled onto the 32 bumps, so a point, two points, a line or a patch on the robot’s fingertip appears as the same pattern on the operator’s.

Pressure mode turns force into area. The peak pressure raises 1, 7 or 19 bumps as it crosses set thresholds: a harder press is felt as a larger patch.

Tracked fingers at work: a Pocky stick lifted into a cup
Inset, top left: the pad raises 7 bumps at each press
what shape
is the contact?
Shape mode: point, two points, line, plane (2×)
and how hard
is it pressing?
peak pressure rises 1 bump 7 bumps 19 bumps at 0.5 N at 3.5 N at 7.5 N
Schematic: 1, 7 or 19 of the 32 bumps light up
Clips: TAG project page · numbers: Sec. III-C, III-D and Appendix A–C of the paper · the bump figure is a schematic with a simplified layout
StartRootsMoveHandsSystem 0SeeSystem 1
46
Results

Force you can feel is force you can control

Lamp brightness: turn the knob, force kept within 0.5–3 N

A light touch, held

12 of 15 with the bumps on. The same glove, bumps off: 4 of 15. The HOMIE glove: 1 of 15.

… and without
looking at all?
Blindfolded: 0.5, 3.5 and 7.5 N held on a spring button

Touch replaces sight

Three force levels, blindfolded: 13 of 15. Bumps off: 3 of 15. The HOMIE glove: 2 of 15.

… and the tracking
on its own?
Lamp colour: press above 3 N while turning

Better fingers help on their own

Bumps off: 12 of 15, the HOMIE glove 9 of 15. Bumps on: 14 of 15 in 8.4 s; the HOMIE glove takes 23.5 s.

< 1°
joint-angle errorbench: within ±0.35° of a servo
0.24°
beside a running PCManus glove: 5.69°, same test
< $500
open-source glovecommercial: over $5,000

Unlocked

Touch on the operator’s fingertip: sub-degree finger tracking and the robot’s contact, in a glove under $500.

Still open

Every robot hand needs its own demonstrations. Can one policy use them all?

Clips: TAG project page (UR5e + XHand) · numbers: Tables II and IV, Sec. III-A and IV-A of the paper · 15 trials per cell · baseline glove: HOMIE
StartRootsMoveHandsSystem 0SeeSystem 1
47
ICRA 2026
Cross-embodiment manipulation

OPFA: one action space for every hand

Describe a hand by its shape, not its joint list: grippers and five-finger hands then share one action latent.
Real UR5e arm · one checkpoint, four end-effectors: grasp the pot and pour
and a task that needs
one finger at a time?
Syringe: an XHand and a Leap Hand press the plunger with the thumb
Before: every hand has its own joint list, so co-training gave each hand its own decoder, trained on that hand’s demonstrations alone. Nothing is shared at the fingers.
From any hand’s joints to one shared action
Hand posejoint angles of one hand
Point cloudits shape, from kinematics
Encodertrained on sampled poses
Action latentwhat the policy predicts
One shared decoderjoints of the mounted hand
One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation. ICRA 2026 · arXiv:2603.14522 · clips: project page
StartRootsMoveHandsSystem 0SeeSystem 1
48
Results

A skill shown on one hand transfers to another

Real UR5e · block into the drawer, on four end-effectors

Four end-effectors, one checkpoint

24 demonstrations per hand and task. Drawer, two-finger gripper: 100 %; with a decoder per hand, 80 %.

… where only the other
hand has data?
traineach hand in its own region Inspire Hand 72 demos XHand 72 demos testin the other hand’s region
Schematic: the test of Table I (simulation tasks)

A skill crosses between hands

Inspire Hand, banana: 98.0 %. XHand, bucket: 94.0 %, against 33.0 % with a decoder per hand, 1.0 % alone.

… and across
many hands and tasks?
Real UR5e · 15 rollouts: tissue, mango, basket, broom

11 end-effectors, 14 tasks

4 on the real arm, 7 more in simulation.
Not uniformly better: with 8 demos each, the UMI gripper gets 27.0 %; alone, 43.0 %.

98.0 %
one shared decoderInspire Hand · banana pick
83.0 %
a decoder per handsame co-training, same test
10.0 %
no co-trainingits own 72 demos only

Unlocked

One action latent for grippers and dexterous hands: a skill shown on one hand transfers to another.

Still open

The legs still know three pedal numbers, and a person writes every command. Three ways out.

Clips: OPFA project page (real robot) · numbers: Tables I, II and Fig. 5 of the paper · the region figure is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
49
The fork

Three roads out of the cockpit

Chapters 2 and 3 leave two problems open. The next three chapters answer them in three ways.
20252026 Decoupled WBCSONIC A VIRALDoorManGRAIL B GR00T N1.x VLAGR00T 2 Turbo C SONIC tokens simulated data
Still openEach skill has its own policy and reward
HOMIE · operator and G1
Still openA person writes every command

One controller for all motion

Chapter 4 · System 0

Can one controller execute any whole-body motion, whoever writes the command?

SONIC · a real G1, running

Give the policy eyes

Chapter 5 · See

Can a policy write the commands from the robot’s own camera?

VIRAL · real G1, RGB in (4×)

Let a model write the commands

Chapter 6 · System 1

One model for every task, on the robot’s own computer?

GR00T 2 Turbo · popcorn
2025The humanoid has one policy per skill and a person in the cockpit. Where does it go from here?
Road ATrain one whole-body controller to follow whatever human motion it is given.
Road BTrain a policy in simulation that writes the commands from the robot’s own camera.
Road CPut one model above the controller: first a VLA, then a world-action model on the robot.
SpoilerThe roads meet in one stack: a world-action model writes tokens for SONIC, and simulation multiplies its data.
StartRootsMoveHandsSystem 0SeeSystem 1
50
Chapter 4  ·  2025 – 2026

System 0

Can one controller execute any whole-body motion,
whoever writes the command?
Decoupled WBC GR00T-WholeBodyControl 2025
SONIC Science Robotics 2026
51
Released Nov 2025
Open source · NVlabs/GR00T-WholeBodyControl

Decoupled WBC: HOMIE’s legs, second version

HOMIE’s split, a learned policy for the legs and no learning for the arms, as the controller used in GR00T N1.5 and N1.6.
HOMIE, 2025 · real G1, teleoperated from the cockpit: squat, grasp the bottle, stand (2×)
and with a VLA
in the operator’s seat?
GR00T N1.6 demo, 2025 · real G1 walks up, squats, takes the bag (blog clip at 2×)
OpenHomie (HOMIE’s repository) · README, News
“there is a homie v2 rl policy at https://github.com/NVlabs/GR00T-WholeBodyControl/tree/main/decoupled_wbc”
NVlabs/GR00T-WholeBodyControl · README
“Decoupled WBC: the decoupled controller (RL for lower body, and IK for upper body) used in NVIDIA GR00T N1.5 and N1.6 models”
Released files · the Balance and Walk networks have the graph of OpenHomie’s deploy.onnx, only wider: 516 inputs → 15 outputs instead of 456 → 12 (our file check)
NVlabs/GR00T-WholeBodyControl, decoupled_wbc/ · News: “[2025-11-12] Initial release with Decoupled WBC for GR00T N1.5 and N1.6.” · clips: HOMIE page, GR00T N1.6 blog
StartRootsMoveHandsSystem 0SeeSystem 1
52
How it works

Seven commands for the legs, IK for the arms

VLA · GR00T N1.630-step action chunks
Upper-body targetsarm · hand · waist joints
7 commandsvelocity ×3 · height · torso ×3
Lower-body policyRL · Balance + Walk · 50 Hz
PD + Unitree G1three-finger hands
legs + waist
arms + hands · not learned · IK in teleoperation
torso orientation
state of all 29 body joints
4 of 7

Two halves, as in HOMIE. A learned policy moves legs and waist from seven numbers, at 50 Hz: three velocities, base height, torso roll, pitch and yaw. Arm and hand targets pass straight through; in teleoperation, IK makes them.

What the second version adds. Read from the released files: HOMIE’s network graph, with three more commands (torso roll, pitch, yaw), three more joints (the waist), and two networks instead of one, Balance and Walk.

A VLA takes the operator’s seat. GR00T N1.6 predicts 30 steps at a time: arm, hand and waist targets, plus a base-height and a navigation command. That is four of the seven numbers; the torso’s three follow from the waist targets.

Other writers, same seven. VIRAL and DoorMan train camera policies on a HOMIE controller that “takes 7 commands” (their repositories). GR00T N1.7 keeps a tag “compatible with the decoupled WBC controller”; its end-to-end workflow is SONIC only.

The limit, measured. In the SONIC paper, HOMIE’s public controller (“OpenHomie”) stays up in 43.0 % of runs (86 of 200) under velocity commands of 0–5 m/s, in simulation (MuJoCo); SONIC stays up in 98.5 %.

GR00T N1.6 demo: real G1 walks up
what did
version two add?
HOMIEV2 commands47 joint targets1215 inputs per step7686 networks12
Released policy files · our reading
and who writes
the commands?
N1.6 demo: sets a mug in the rack
who else writes
these seven numbers?
Real G1: VIRAL (left), DoorMan · RGB in, commands out

Unlocked

Legs a VLA can drive with seven numbers: the controller used in GR00T N1.5 and N1.6.

Still open

Two halves, a velocity command: version one survives 43.0 % of SONIC’s test. What if one policy tracked any human motion?

Clips: GR00T N1.6 blog; VIRAL and DoorMan pages · interface: decoupled_wbc code, Isaac-GR00T configs · 43.0 % vs 98.5 %: SONIC paper, Fig. 2I · table, “4 of 7”: our reading of the released files
StartRootsMoveHandsSystem 0SeeSystem 1
53
Science Robotics 2026 · arXiv Nov 2025

SONIC: one controller
for any human motion

One policy · a real Unitree G1
Motion tracking, scaled up
54
The bet

Scale three axes, keep the task simple

Real G1 · the task itself: copy a person, frame by frame
S0one trackerfor any motion
Commandtokensnext slide
Data≈ 700 hoursof mocap
Datamotion capture
100M+frames · 611 h at 50 Hz
4M10M22M100M+frames
Modelnetwork size
42Mparameters · up from 1.2M
1.2M16M42Mparameters
ComputeGPUs in parallel
128 GPUs7 days ≈ 21k GPU hours
1632128 GPUs1 column =8-GPU node
Unseen motionsabsent from training
99.6 %success · 98.0 % at 1.2M
1.2M model27.7 mm42M model23.8 mmtracking error on unseen motions
One task, no reward per skill. The robot only has to follow a reference motion, frame by frame, so every frame of motion capture is a training target. The bet: leave the task alone, and scale the data, the model and the compute around it.
Data. About 700 hours of human motion capture are retargeted to the G1. Motions the robot cannot execute, such as stair climbing and seated activities, are filtered out: 611 hours remain, 100M+ frames at 50 Hz in 317,189 clips.
Model. Three sizes are trained: 1.2M, 16M and 42M parameters. Every real-robot experiment uses the 42M model; on the G1’s onboard Jetson Orin one forward pass takes 1–2 ms.
Compute. 16, 32 and 128 GPUs, each run to 50k iterations with 4,096 simulated robots per GPU: about 2k, 9k and 21k GPU hours. The largest run is 128 GPUs for 7 days.
Tracking keeps improving with scale, most on motion types that never appear in training: there, success in simulation rises from 98.0 % with the 1.2M model to 99.6 % with the 42M model, and the tracking error falls from 27.7 to 23.8 mm.
SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. Science Robotics 11(117), 2026 · clip: project page · Fig. 2
StartRootsMoveHandsSystem 0SeeSystem 1
55
How it works

Every command becomes the same token

Human motiona person’s 3D joints
Robot motionthe robot’s own joints
Hybrid motion3 points + robot legs
Human encoder
Robot encoder
Hybrid encoder
Universal tokenFSQ: 2 × 32 numbers
Control decoder→ 29 joint targets
PD + Unitree G1control at 50 Hz
Motion decodertraining only
Auxiliary lossestie the encoders
PPOone training stage
Tracking rewardevery frame a target
proprioception
token only
update

From motion to token

What should the body do next? A reference motion says it: the next 10 frames, 0.1 s apart.

An encoder squeezes them into 2 × 32 numbers, each rounded to one of 32 levels. These 64 numbers are the whole command.

Real G1 · tracking a boxing motion
Reference motion · joint angles over timenext 10 frames · 0.1 s apartnow+ 0.9 sRobot encodertoken 1 · 32 numberstoken 2 · 32 numbers321= one token: 64 numbers, a new one every 20 msOne of the 64 numbers141720encoder says 17.38kept: level 17 of 32FSQ: no codebook

One decoder moves the body

The control decoder reads the token and the robot’s own sensing over the last 10 steps: joint positions and velocities, rotation rate, gravity direction, previous action.

Out come 29 joint targets, 50 times a second. Balance is its job, not the writer’s.

Real G1 · one foot on the pedal while the arms reach
Real G1 · crawling on hands and knees

Three formats, one token

The same motion can arrive in two more formats: a person’s 3D joints, or only head and two hands on top of the robot’s leg motion.

Each format has its own encoder. All three write the same 64 numbers.

Robot motionHuman motionHybridnow+0.9 snow+0.9 snow+0.9 sjoint positions + velocities10 frames, 0.1 s apartfrom a motion clip or the plannerRobot encoder3D joints of a person (SMPL)10 frames, 0.02 s apartfrom VR trackers, video, text, musicHuman encoderhead + two hands, right now+ robot legs: 10 frames, 0.1 s apartfrom a headset and two controllersHybrid encoderone token: the same 64 numbers, whichever encoder wrote them

Trained to agree

PPO on a tracking reward trains encoders and decoder together, in one stage.

Three more losses tie the encoders: one motion must give one token in every format, and decode back to the robot motion. Without them the tokens sit about 8× further apart.

$\mathcal{L} =$
$\mathcal{L}_{\text{ppo}}$follow the reference
+
$\mathcal{L}_{\text{recon}}$any token decodes to the robot motion
+
$\mathcal{L}_{\text{token}}$three formats, one token
+
$\mathcal{L}_{\text{cycle}}$human → robot → the same token
Distance between the three encoders’ tokenswith the losses0.57without4.23mean L2 distance on a crawling motion · Fig. S7Tracking unseen motions through each encoderrobot23.8 mm· 99.6 %human24.4 mm· 99.6 %hybrid26.5 mm· 99.2 %tracking error and success rate · Table 3C
Clips: SONIC project page · token and format figures: schematics, illustrative values · numbers: Fig. S7 and Table 3C of the paper
StartRootsMoveHandsSystem 0SeeSystem 1
56
The interface

One token, many writers

Universal token2 × 32 = 64 numbers per control step · all the controller is told about the motion
control decoderUnitree G129 joint targets, 50 times a second. The same controller under every writer.
Gamepad + planner$\mathcal{E}_r$

A gamepad

A kinematic planner turns speed, heading and style into the next 0.8–⁠2.4 s of robot motion, and replans up to 10 times a second.

Gamepad + planner: boxing
VR teleoperation$\mathcal{E}_h$ , $\mathcal{E}_m$

A person in VR

Headset, controllers and ankle trackers give the operator’s whole body. Or only head and wrists: the planner adds the legs.

a person, live?
Whole-body VR · operator at the back
Video · text · music$\mathcal{E}_h$

A camera, a prompt, a song

GEM, a human-motion model, estimates motion from a camera or generates it from text or music. The human encoder reads it directly.

no hardware on the person?
Camera → GEM: person and robot crawl
GR00T N1.5 VLAno encoder

A vision-language-action model

It writes the token itself in 4 of 5 real tasks: 75 % average success over all 5. With body poses instead, on 3 of them: 27 % against 68 %.

no person at all?
VLA: drill to bin, bin to shelf (2×)
Whoever can write 64 numbers can drive the whole body. The controller underneath is never retrained.
Clips: SONIC project page (nvlabs.github.io/GEAR-SONIC) · planner: Sec. 2.2 of the paper · VLA success rates: Table 1
StartRootsMoveHandsSystem 0SeeSystem 1
57
The interface · planner

The planner writes motion; the tracker keeps balance

Gamepadmode, speed, direction
Kinematic plannerposes only, no physics
Robot motionthe next 0.8–⁠2.4 s
SONIC controllertoken → 29 joint targets
PD + Unitree G1control at 50 Hz
4 frames of its own plan
proprioception
Gamepad + planner: a stealth walk

A second, generative model

It turns mode, speed and direction into whole-body robot motion. It learned from the same motion capture as the tracker.

what does it compute?
4 recent framestarget posegenerated in betweennow+ 0.8 s+ 2.4 s
Schematic: one plan fills in the gap

It fills in the gap

From 4 recent frames of its own plan to a target pose 0.8–⁠2.4 s ahead. The planner picks the length.

who keeps the balance?
Gamepad + planner: an elbow crawl

No physics inside

It continues from its own plan, not from the robot’s sensors. Keeping the body on that plan, and upright, is the tracker’s job.

fast enough to steer?
Gamepad + planner: squat, kneel, stand up

Fast enough to steer

It replans up to 10 times a second, or at once on a new command. One plan: ≈ 12 ms on the Jetson Orin, under 5 ms on a laptop.

25+ skills and styles from one planner, each defined by a single example clip. Nothing is retrained to add one.
Clips: SONIC project page (nvlabs.github.io/GEAR-SONIC) · planner: Sec. 2.2, 3.3, 3.6 and Supp. S1.2 of the paper · the timeline is a schematic
StartRootsMoveHandsSystem 0SeeSystem 1
58
The interface · teleoperation

Drive it from VR, record the demonstrations

Universal tokenthe same 64 numbers, however the operator is tracked
Headset · controllers · ankle trackers$\mathcal{E}_h$
Whole-body VR · operator and robot kneel together

Whole-body mode

The operator’s pose is the command: 87 numbers per frame, body and hands. The human encoder reads the motion; no retargeting at run time.

Headset · controllersplanner + $\mathcal{E}_m$

3-point mode

Only head and both wrists are tracked; the operator steers the walk. The planner writes the legs, and the hybrid encoder reads both.

without the ankle trackers?
3-point VR · one hand carries, one presses the lift button
and data to teach a model?
Demonstrations per VLA taskteleoperated · counts as printed in Table 1AApple to plate3-point mode · single object300Object pickup13 objects × 3003,900Open trash can with a foot200Soda can to trash canmulti-object1,000Drill and box relocation300bars to scale
Both modes write the same token, so demonstrations from either can share one action space for a VLA to learn.
Clips: SONIC project page (nvlabs.github.io/GEAR-SONIC) · modes: Sec. 2.4 and Supp. S1.4 of the paper · counts: Table 1A and Sec. 2.5
StartRootsMoveHandsSystem 0SeeSystem 1
59
The interface · VLA

A VLA writes the tokens, feet included

Paper · GR00T N1.5, fine-tuned · real Unitree G1
VLA: one foot presses the pedal, the lid opens

It writes the token itself

Fine-tuned on the teleoperated demonstrations. In 4 of the 5 tasks its action is the 64-number token plus 14 hand joints: no encoder, the whole body, feet included.

The hardest task chains five skills, from grasping a can to pressing the pedal and dropping it in.

how often does it succeed?
Five real tasks · success10–20 trials per task · no partial credit · Table 1token3-point commandsbody poses insteadApple to plate90 %Object pickup · carrot75 %Object pickup · scrub95 %Open trash can · foot70 %Soda can to trash can60 %Drill + box relocation70 %60 %20 %0 %Average of the 5 tasks75 %On the 3 tasks tested with poses68 % vs 27 %
and outside the paper?
Isaac-GR00T example · N1.7 + SONIC: a towel pickup
Open source · GR00T N1.7 + SONIC

Repo docs · workflow defaults

The VLA runs at 2.5 Hz and sends 40 tokens per call; SONIC plays them at 50 Hz.

Isaac-GR00T example · real G1

159 of 216 table pickups (73.6 %) and 8 of 12 soda cans to the bin. Objects seen in training; retries count.

The VLA decides what to do; SONIC handles how: balance, walking and whole-body coordination.
Clips: SONIC project page · Isaac-GR00T example · numbers: Table 1 of the paper · rates, callout: GR00T-WholeBodyControl docs · 159 / 216, 8 / 12: Isaac-GR00T example README
StartRootsMoveHandsSystem 0SeeSystem 1
60
Under the hood

SONIC: Three encoders, one 64-number FSQ token, one decoder for 29 joints

On the robot: nothing is retrained
Auxiliary losses$\mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{token}} + \mathcal{L}_{\text{cycle}}$
Motion decoder $\mathcal{D}_r$MLP 2048-⁠1024-⁠512-⁠512 training only → robot motion
PPOcritic sees privileged state ${\gamma = 0.99}$ · clip 0.2
Tracking reward $r_t$7 tracking terms, 5 penalties heaviest: head, wrists, ankles
Motion dataset611 h of mocap, retargeted to the G1 · 317,189 clips
Encoders $\mathcal{E}_r, \mathcal{E}_h, \mathcal{E}_m$MLP 2048-⁠1024-⁠512-⁠512 each robot · human · hybrid format
Universal token $z$FSQ: 2 tokens × 32-⁠d, 32 levels = 64 numbers, every 20 ms
Control decoder $\mathcal{D}_c$MLP 4096-⁠4096-⁠2048-⁠2048-⁠ 1024-⁠1024-⁠512-⁠512 → 29-⁠d
PD + Isaac LabUnitree G1 · 29 joints control at 50 Hz
Motion writersplanner → $\mathcal{E}_r$ · GEM, VR → $\mathcal{E}_h$ 3-⁠point VR + planner → $\mathcal{E}_m$
Token writer: a VLAGR00T N1.5, fine-⁠tuned → 64-⁠d token + 14-⁠d hands
10 frames
round
targets
proprioception, last 10 steps
update $\mathcal{D}_c, \mathcal{E}$
sees only $z$
$\hat g_r$
target $g_r$
update $\mathcal{E}, \mathcal{D}_r$
a motion
no encoder
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
Tracking loop: motion → token → joint targets, state backPPO on a dense tracking reward trains the whole pathThree losses through a motion decoder tie the encodersOn the robot: write a motion, or write the tokenThe cost, and the choice of quantizer
Objective
  • $r_t$: 7 tracking terms + 5 penalties; heaviest $2.0\,\exp\!\big(-\|p^{\text{ee}}_t - \hat p^{\text{ee}}_t\|^2 / 0.1^2\big)$ on head, wrists, ankles
  • $\mathcal{L}_{\text{ppo}}$ updates $\mathcal{E}$, FSQ, $\mathcal{D}_c$ and the critic; only the critic sees privileged state
  • $\mathcal{L}_{\text{recon}} = \sum_{k \in \{r,h,m\}} \|\mathcal{D}_r(z_k) - g_r\|^2$, $\quad\mathcal{L}_{\text{token}} = \|z_r - z_h\|^2 + \|z_r - z_m\|^2 + \|z_m - z_h\|^2$
  • $\mathcal{L}_{\text{cycle}} = \|\mathcal{E}_r(\mathcal{D}_r(z_h)) - z_r\|^2$; $\ \mathcal{L} = \mathcal{L}_{\text{ppo}} + \mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{token}} + \mathcal{L}_{\text{cycle}}$, one stage
Training recipe
  1. Retarget ≈ 700 h of mocap to the G1; drop stairs and sitting: 611 h remain
  2. Sample 1 s bins by capped failure rate; randomize physics, pushes, targets
  3. Encode each motion in three formats; optimize all four losses in one stage
  4. Deploy the 42M model on the G1’s Jetson Orin (TensorRT); no retraining per writer
611 h100M+ frames · 317,189 clips
128 GPUs7 days ≈ 21k GPU hours
4,096environments per GPU
1–2 msper forward pass, onboard Jetson Orin
The detail that makes it work

FSQ only rounds numbers: no learned codebook that can collapse, no commitment loss, and straight-through gradients that train together with PPO.

Versus its predecessor

vs HOMIE’s controller (RL legs on a velocity command, arms set directly): one policy for all 29 joints, commanded by a token. One stage, no teacher.

StartRootsMoveHandsSystem 0SeeSystem 1
61
Results

The generalist outlasts the specialist

Real G1 · a motion generated from a text prompt

Motions it never trained on

99.6 % of 7,016 clips from motion types absent in training are tracked in simulation; 98.0 % with the 1.2M model.

… and on the real robot?
Real G1 · a swinging load hits it, and it keeps standing

The same policy on hardware

123 of 124 motions succeed on the real G1, one trial each; all 124 in simulation. Error: 25.7 mm real, 22.3 mm simulated.

… against a controller built
for one job?
Real G1 · running on a gamepad command

Against a specialist

Velocity commands of 0–5 m/s in simulation: 98.5 % of 200 runs stay up, against 43.0 % for OpenHomie, HOMIE’s controller.

98.5 %
SONICunseen motions · MuJoCo test
82.0 %
BeyondMimicsame MuJoCo test · LaFAN data
61.6 %
Any2Tracksame MuJoCo test · LaFAN data

Unlocked

One controller for almost any human motion on the real robot, and a 64-number token anything upstream can write.

Still open

The controller is blind, and someone must write the commands. Can a policy do it from the robot’s own camera?

Clips: SONIC project page (nvlabs.github.io/GEAR-SONIC) · numbers: Sec. 2.1 and Fig. 2 of the paper · the tracker comparison is not data-matched
StartRootsMoveHandsSystem 0SeeSystem 1
62
Chapter 5  ·  2025 – 2026

See

Can a policy write the commands from the robot’s
own camera, trained only in simulation?
VIRAL CVPR 2026
DoorMan CVPR 2026
GRAIL CoRL 2026 · Oral
63
CVPR 2026

VIRAL: from pixels
to loco-manipulation

One RGB camera · a real Unitree G1 (4×)
Trained only in simulation
64
How it works

Command the controller, do not replace it

Teacherfull state · PPO
Staged reward5 stages · 36 terms
200 sim demosteleoperated
RGB studentsees pixels · 50 Hz
Commandvelocity · arms · hand
HOMIE controllerchapter 3 · frozen
PD + simulatorIsaac Lab · PD 200 Hz
Real Unitree G1no fine-tuning
PD targets
11 increments
reward
resets
the same
imitate
one RGB image + proprioception
match
zero-shot

Keep the controller from chapter 3. HOMIE’s policy, frozen, turns a command (walking velocity, height, arm joints; here also fingers) into joint targets at 50 Hz. In chapter 3 a person wrote that command.

A teacher writes the command. PPO on the full simulator state; its output is 11 increments, accumulated into the command. Every episode starts from a snapshot drawn from 200 teleoperated simulation demos.

A student that sees pixels imitates it. One 108 × 192 RGB image plus proprioception in. Half the environments follow the teacher, half the student (DAgger); light, materials and camera pose are randomized.

Match the hand and the camera. The high-gear-ratio fingers are system-identified from a real grasp-and-release recording; the camera view is aligned to real images. Then the student runs on the real G1, zero-shot.

HOMIE: a person writes the command
and who writes
the command now?
Teacher success, simulation full method nearly 95 % no demo resets below 10 % absolute commands ≈ 0 %
Schematic: one ingredient removed
and without
the simulator’s state?
Sim, time lapse · inset = the input
will it survive
the real robot?
Finger SysID, before | after: simulated over real hand
VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation. CVPR 2026 · clips: project pages · bars: Sec. 4.1; ≈ 0 % read off Fig. 9 · 36 terms: our count
StartRootsMoveHandsSystem 0SeeSystem 1
65
How it trains

Scale is part of the recipe

Sim · the teacher policy: every robot is one environment

Teacher: 16 GPUs, no rendering

PPO on the full simulator state in 32,768 parallel environments: two nodes of eight L40S GPUs, 2,048 on each GPU.

and with
fewer GPUs?
Teacher success rate 90 % 50 % 0 0 2 4 6 8 10 h 8–16 GPUs 4 GPUs 1–2 GPUs
Traced from Fig. 14 · teacher success, simulation

Too few GPUs, no teacher

With 1–2 GPUs the teacher plateaus far below; 8–16 GPUs take it above 90 % success. More compute is not only faster.

and when every step
has to be rendered?
Sim · the rendered camera view; inset = augmented

Student: 64 GPUs, all rendered

One 108 × 192 image per environment and step, 1,024 environments per GPU. Rendering typically costs 10× or more in throughput.

≈ 7 h
64 GPUs reach 85 %student success · read off Fig. 15
15–20 h
1 GPU reaches 85 %read off the same plot · noisier
86–88 %
every run after 50 h1 to 64 GPUs · read off Fig. 15
Not only faster. For the teacher, large-scale simulation is “not only beneficial but often necessary” (the paper). The student ends about as high on 1 GPU; 64 GPUs get there in under half the time.
Clips: code repository README (github.com/NVlabs/GR00T-VisualSim2Real), VIRAL project page · numbers: Secs. 2, 4.7–4.8, 7.1, Tables 5–6 · curves traced from Fig. 14 · tiles read off Fig. 15
StartRootsMoveHandsSystem 0SeeSystem 1
66
How it transfers

Match the hand and the camera

One command for the hand. The G1’s high-gear-ratio fingers move differently in simulation. So the policy sends one open / close command, and finger armature, stiffness and damping are fitted to a real grasp-and-release.

Align the camera, then jitter it. Intrinsics follow the manufacturer’s specification; extrinsics are found by matching a rendered image to a real one. Training then shifts the camera by up to ± 2–5 cm and ± 0.05–0.1 rad.

Randomize what cannot be matched. Light, materials, image quality and camera delay vary during training. Trained with none of it, the student keeps 0.649 of its success in simulation; of the single items, camera pose matters most.

Let the student make its own mistakes. Half the environments act with the teacher’s action, half with the student’s. Teacher-only rollouts reach the lowest loss and a success rate of 0.00: loss is not success.

Real G1 · testing the finger primitive
and what the
camera sees?
Paper figure (Fig. 6), panels stacked
and what cannot
be matched?
Student success, simulation randomization left out in training: nothing left out 1.000 materials 0.947 dome light 0.947 camera pose 0.893 everything 0.649 normalized · 200 episodes per bar
Schematic after Fig. 13 · simulation
and the student’s
own mistakes?
Teacher’s share of the rollouts bar = success rate 0 % · student only 0.80 loss ≈ 0.19 25 % 1.00 loss ≈ 0.17 50 % · the default 1.00 loss ≈ 0.14 75 % 0.30 loss ≈ 0.11 100 % · teacher only 0.00 loss ≈ 0.075, the lowest
Schematic after Fig. 11 · loss: read-off
Clip: VIRAL project page (“Our Visual Sim2Real Journey”, 28 July 2025) · one hand command: page, Table 3, released code · numbers: Sec. 2.3, Table 7, Figs. 11 and 13 · loss read off Fig. 11
StartRootsMoveHandsSystem 0SeeSystem 1
67
Under the hood

VIRAL: A privileged teacher, an RGB student, and HOMIE’s frozen controller underneath

DAgger + BC lossMSE to the teacher’s action · $\alpha = 0.5$
Teacher $\pi_{\text{teacher}}$MLP 512-256-128 · 226-d privileged state
PPO$\gamma = 0.998$ · 32,768 environments
Staged reward $r_t$5 stages · 36 terms (our count)
Visual randomizationdome light, materials, camera pose, image, delay
Student $\pi_{\text{student}}$128‑d image feature + 113‑d state · 50 Hz
WBC commandvelocity · height · arm and finger joints
HOMIE WBC $\pi_{\text{wbc}}$pretrained, frozen · 50 Hz
PD + Isaac LabPD controller at 200 Hz
Real Unitree G1RealSense D435i RGB · no fine-tuning
Real-to-sim alignmentfinger SysID · camera intrinsics + extrinsics
200 sim demosteleoperated · state snapshots (RSI)
$a_t$ · 11-d
PD targets
RGB 108 × 192 + proprioception
increments $a_t$
update
$r_t$
privileged state
resets
labels
update
RGB
recordings
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: a pixel policy writes HOMIE’s command at 50 HzPhase 1: a privileged teacher learns the task with PPOPhase 2: the RGB student imitates it under visual randomizationSim-to-real: match fingers and camera, then deploy zero-shotWhat it costs, and what makes it work
Objective
  • Action: $a_t = (\Delta v_t,\ \Delta\omega^{\text{yaw}}_t,\ \Delta q^{\text{arm}}_t,\ \Delta q^{\text{finger}}_t)$, 11 numbers, accumulated into the WBC command
  • Teacher reward: $r_t = \sum_{i=0}^{4} w_i\,\mathbb{1}[s_t = i]\,r^{(i)}_t$, stages: walk, pre-place, place, grasp, turn
  • e.g. $r_{\text{walk}} = \exp\!\big(-4\,(\|p_{\text{robot}} - p_{\text{GraspObj}}\| - 0.45)^2\big)$, $\ r_{\text{turn}} = -|y_{\text{robot}} - y_{\text{desired}}|$
  • $\mathcal{L}_{\text{distill}} = \mathbb{E}_{o_t \sim \rho_o}\|\pi_{\text{teacher}}(o^{T}_t) - \pi_{\text{student}}(o^{S}_t)\|_2^2$, $\ \rho_o = \alpha\,\rho^{T}_o + (1-\alpha)\,\rho^{S}_o$, $\ \alpha = 0.5$
Training recipe
  1. Collect 200 teleoperated demos in simulation; reset every episode to a random snapshot
  2. Train the teacher with PPO on 16 L40S GPUs, 32,768 environments, no rendering
  3. Distill on 64 GPUs with tiled rendering; teacher and student each roll out half
  4. Deploy with no fine-tuning: sim fingers and camera were matched to the real robot
200teleoperated simulation demos
16 GPUsteacher · 32,768 environments
64 GPUsstudent · tiled rendering
0real-world demonstrations or fine-tuning
The detail that makes it work

Demo resets and incremental commands. Without the resets the teacher stays below 10 % success (nearly 95 % with them); with absolute commands it fails.

Versus its predecessor

vs HOMIE (chapter 3): the same frozen controller and command, but a simulation-trained RGB policy writes the command instead of a person in a cockpit.

StartRootsMoveHandsSystem 0SeeSystem 1
68
Tray · 15 cm beyond the table edge
Robot start · “far” from the table (2×)
and when the table
changes?
Table height · 80.7 cm, the highest of eight
Table cloth · red, one of eight colours
and a new
object?
Object · a bowling pin, one of 13
and when
it fails?
Failures · a fall | an apple it cannot lift
VIRAL · real robot

Positions. Tray from 20 cm inside the table to 15 cm beyond its edge; robot start near, middle or far.

The table. Eight heights from 66.5 to 80.7 cm, eight cloth colours, three table tops.

Objects and light. 13 everyday objects, here a bowling pin; bright, flashing and dark light.

And it fails, too. A fall, a stuck hand, a dropped object, three unfamiliar objects it cannot lift.

Change the scene, keep the policy

Videos, not statistics. Ten kinds of scene change, 55 clips on the project page (our count). The paper’s sentence for all of them: “VIRAL consistently completes the task without additional tuning.” No success rate is printed.
StartRootsMoveHandsSystem 0SeeSystem 1
69
Results

Zero-shot on the real robot, at an expert’s pace

Real G1 · time lapse of a 54-cycle run

54 of 59 trials succeed

Consecutive real-world trials of the whole loop, with no real-world training data. That is 92 %; an expert teleoperator reaches 100 %.

20.2 s
per cycle, VIRALzero-shot · 92 % success
21.4 s
expert teleoperatorover 1000 h experience · 100 %
47.0 s
non-expert teleoperatorabout 1 h experience · 73 %

Unlocked

A walk-place-grasp-turn loop from pixels: learned only in simulation, zero-shot on the real G1, at an expert’s pace.

Still open

One task, two tables, a hand-designed staged reward. Does it hold for articulated objects it has never seen?

Clip: VIRAL project page (viral-humanoid.github.io) · numbers: Sec. 3.1 and Fig. 7 of the paper (real robot)
StartRootsMoveHandsSystem 0SeeSystem 1
70
CVPR 2026
Articulated objects from RGB

DoorMan: open doors it has never seen

Randomize the door, its textures and the light in simulation, until a real door is just one more sample.
Real Unitree G1, autonomous (1×) · top right: the policy’s RGB input
and a door that opens
towards the robot?
Real G1 · a pull door: press the lever, pull, step around, walk out
Before · measure the door first
“Where is the handle, in 3D?”
depth sensingobject localizationscripted motion primitives
DoorMan · pixels to action
“What do I do, given this image?”
RGB only, no depthsimulation data onlyzero-shot on real doors
Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer. CVPR 2026 · clips: project page
StartRootsMoveHandsSystem 0SeeSystem 1
71
How it works

Teacher, student, then learn from its own rollouts

PPOsix-stage reward
Teachersees simulator state
Reset buffer100 states per stage
GRPOsuccess or failure
StudentRGB + proprioception
HOMIE controllerpretrained, reused
G1 at a doorIsaac Lab → real robot
Door generatorshape, texture, light
update
commands
targets
scene
cache
load
labels
commands
its own rollouts
update

A teacher that sees everything. PPO on simulator state: door and handle poses, contact forces on the hands. A six-stage reward leads it from walk-up to walk-through; HOMIE’s pretrained controller handles the legs. 1 GPU, 6 hours.

Restart where it got stuck. Entering a new stage caches the simulator state, and a reset may load such a snapshot instead of the start. The reward is untouched; the teacher simply gets to practise the late stages.

A student with only a camera. DAgger: the student acts from RGB and proprioception; the teacher labels every state it reaches. Each environment has its own door; textures and 5,233 dome lights vary its look. 32 GPUs, 24 hours.

Then it learns from its own rollouts. GRPO, an actor-only PPO, fine-tunes the student on a reward that is mainly success or failure. It finds what the teacher never showed, such as keeping the object in view. 64 GPUs, 12 hours.

Teacher rollout, no textures
and if it never
grasps the handle?
walk upgraspwalk through012345cacheloadreset bufferwith it: every stage by ≈ 1,700 iterationswithout: stuck before stage 2, the grasp
Schematic: resets reload cached states of later stages
the same task,
from pixels?
Same rollout: materials and light vary
and what the teacher
never showed?
teacher 80–90 %80.8–85.8 %after GRPO50–70 %after DAgger0GRPO iterations200success rate in simulation
Schematic: student success in GRPO
Clips: project page · numbers: Sec. 2.2, 2.4, 3.3 and 3.4 of arXiv 2512.01061 · compute (L40S GPUs): project page · both figures are schematics
StartRootsMoveHandsSystem 0SeeSystem 1
72
How it trains

Randomize the pixels, not only the door

Door generator5 types · 3 categories
Materialson every surface
Dome lights5,233 + Omniverse’s
RGB studentsees only the render
0.8–1.1 m1.9–2.2 mtallhandle at0.85–0.95 m80–120kgspring, latchmodelleda new draw per environment
Schematic: ranges of Table 3
and how much do
the pixels matter?
no randomizationsolid colours10 % of texturesall texturessuccess on unseen doors, in simulation (%)10.867.579.285.8push leverthe simplest5.065.877.580.8pull leverthe hardest20.070.077.585.0push barneeds force73.355.876.7without dome lights
Table 1 · 120 unseen-door trials each · bars 2–4 with dome lights
Every environment gets its own door. Panels 0.8–1.1 m wide, 1.9–2.2 m tall, 80–120 kg; stiffness and damping of hinge and handle are drawn at random. The latch is modelled: the door gives way only once the handle is down.
Every surface gets a material. Trained on one grey look, the student opens 5–20 % of unseen doors. Solid colours under random dome lights, the classic recipe, reach 65.8–70 %; a tenth of the PBR textures 77.5–79.2 %; all of them 80.8–85.8 %.
The sky changes too. A dome light sets lighting and background: Omniverse’s own, plus 5,233 from Poly Haven, indoors and outdoors, at various times of day. Without them the full recipe drops to 73.3, 55.8 and 76.7 %.
No real scene is copied. The generator is not biased towards the size, physical response, texture, colour or light of any real location, and every real scene in the evaluation is unseen. To the student, a real door is one more sample.
Isaac Lab fly-over: one door per environment
and one rollout,
in three looks?
Simulation: one rollout, the look keeps changing
Clips: project page; the lower one is its scroll animation at 30 fps (our choice) · numbers: Tables 1 and 3, Sec. 2.4, App. B of arXiv 2512.01061 · Table 1 as simulation: our reading
StartRootsMoveHandsSystem 0SeeSystem 1
73
Under the hood

DoorMan: A staged-reset teacher, an RGB student, then GRPO, all on HOMIE’s controller

DAggerteacher labels the states the student reaches
Teacher $\pi_T$sees door and handle poses, hand contact wrenches
PPOsix-stage reward · 33 terms (our count)
Reset bufferlast 100 snapshots of each stage
Door generatora door per environment · materials · dome lights
Student $\pi_S$ResNet → LSTM 2 × 512 → MLP · at 50 Hz
Command3 velocities · 14 arm joints · 2 hand values (released code)
HOMIE controllerpretrained, frozen · moves legs and waist
PD + Isaac LabUnitree G1 at a generated door
GRPOactor-only PPO: no value network
Group of $G$ rolloutsreturn $R_i$: mainly success or failure
$a_t$
PD targets
one RGB image + proprioception
commands
update
labels
update
RGB
its own rollouts
$\hat A_i$
update
cache
load
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: one RGB image in, a command out, at 50 HzPhase 1: a privileged teacher, PPO, and resets from later stagesPhase 2: DAgger distills the teacher into the RGB studentPhase 3: GRPO fine-tunes the student on its own rolloutsWhat it costs, and what makes it work
Objective
  • Action (released code): $a_t = (\Delta v_x,\ \Delta v_y,\ \Delta\omega_z,\ \Delta q^{\text{arm}}_{1:14},\ \Delta u^{\text{hand}}_{1:2})$, 19 numbers
  • Teacher reward: 33 terms by stage, e.g. stages 1–3: $-15.0\,\|u_{\text{HOMIE},[0:3]}\|_2$ (stand still)
  • Staged reset: $\tilde\rho_\alpha = \sum_{y} \alpha_y\,\rho_y$, $\ \sum_{y} \alpha_y = 1$: a share $\alpha_y$ of rollouts starts in stage $y$
  • GRPO: $\hat A_i = (R_i - \operatorname{mean}(R))/\operatorname{std}(R)$ for every step of rollout $i$, in PPO’s clipped loss
Training recipe
  1. Train the teacher with PPO on privileged state and a reward in six stages
  2. Cache the state at every new stage; resets may load one of the last 100 snapshots
  3. Distill into the RGB student with DAgger; each environment has its own door and look
  4. Copy the student; fine-tune it with GRPO on a reward that is mainly success or failure
33reward terms in six stages (our count)
100snapshots kept per stage for resets
1,542 hL40S GPU time, three phases (our sum)
50 Hzpolicy rate · off-board RTX 4090
The detail that makes it work

Staged reset. With 100 snapshots per stage the teacher reaches all stages by ≈ 1,700 iterations; with 10 it takes over 4,000; with none it stalls before the grasp.

Versus its predecessor

vs VIRAL: the same teacher-student recipe, again on HOMIE’s controller. New: resets from the task’s own stages, not teleoperated demos, and a GRPO phase.

StartRootsMoveHandsSystem 0SeeSystem 1
74
Orange door · top right: the policy’s RGB input
Meeting-room door · filmed from a camera on the robot
Second location · grey office door
the same door,
with another handle?
Mock-up door, silver lever · camera on the robot
Same mock-up door · black lever
and when it
does not work?
Failure · the hand falls short, the door stays shut
DoorMan · real robot

Installed doors. Two locations, each pushed open from one RGB stream: the inset marked “Policy Input”.

Same door, new handle. A mock-up door with a silver, a brass or a black lever; two of the three are shown.

And a miss. The hand stops short, then touches only the lever’s tip. The page names three failure patterns, no rates.

Real doors it never trained on

5 + 1
doors on video5 installed, 1 mock-up · our count
10 : 1
pushed : pulled open11 success clips · our count
0
real push-bar doors on videoone of 3 evaluated categories
Clips: project page (doorman-humanoid.github.io), as published, not sped up · every clip shows a real Unitree G1 · counts: our count of the page’s 17 real-robot clips
StartRootsMoveHandsSystem 0SeeSystem 1
75
Results

As reliable as an expert teleoperator, and faster

Policy and people drive the same Unitree G1 through the same whole-body controller; every real door is unseen.
Real G1 · mock-up door, walls draped in blue and yellow
83 %

success on real, unseen doors, zero-shot. A person teleoperating the same robot: 80 % (expert), 60 % (non‑expert).

and against a person
at the controls?
Teleoperation baseline: this attempt ends in a fall
15.40 s

average, from 1 m in front to 1 m beyond. A person teleoperating: 20.02 s (expert), 22.55 s (non‑expert).

and which ingredient
makes it generalize?
Isaac Lab: every training door looks different
85.8 %

in simulation, with textures and dome lights randomized. With neither: 10.8 % (push lever, 120 trials each).

Unlocked

Unseen real doors, from RGB alone: zero-shot from simulation, and faster than a person teleoperating the same robot.

Still open

One skill, a hand-built scene, a six-stage reward. Where does the data for the next thousand skills come from?

Clips: project page (doorman-humanoid.github.io) · numbers: Fig. 5 and Table 1 of arXiv 2512.01061 (same values in the CVPR 2026 version) · Table 1 as simulation: our reading
StartRootsMoveHandsSystem 0SeeSystem 1
76
CoRL 2026 oral · Best Paper Award Finalist

GRAIL: robot data
from a video world model

Simulated Unitree G1s
Every motion started as a generated video
77
How it works

3D first, then video, then physics

Known 3D scenemesh · camera · scale
Video world modelone frame + a prompt
4D reconstructionhuman + object
Retargetto the Unitree G1
RL trackerin simulation
RGB policycamera in, tokens out
SONIC · whole-body controllerpretrained · turns tokens into joint targets
known: camera · scale · object mesh
known: a character of the robot’s size
built on
tokens

The scene exists before the video. Object mesh, room, camera and a robot-sized character are set in 3D. One frame is rendered, and a video model rolls out the interaction from it, with a static camera.

So the video can be lifted back to metric 4D. Human and object are reconstructed against the known camera, depth and mesh, then retargeted to the G1: the character already has its proportions.

Then physics decides. An RL tracker built on SONIC has to execute each motion in simulation, object included. What it executes becomes robot data; a retarget alone sinks into the floor.

The data trains a policy that sees. The trackers are distilled into egocentric RGB policies in randomized simulation. They write SONIC tokens: the interface of the last chapter.

Known 3D is what pays. Without the depth loss, which is anchored to the scene’s rendered depth, downstream tracking success falls from 81.4 % to 42.6 % (124 motions, 43 objects).

Generated video · a known 3D room
does the motion
survive physics?
Simulated G1 · left: kinematic retarget, feet sink · right: tracked in physics
and what does
a policy see?
Simulation · rendered head-camera RGB
GRAIL: Generating Humanoid Loco-Manipulation from World Models. CoRL 2026 · clips: project page and oral deck · number: Table 8
StartRootsMoveHandsSystem 0SeeSystem 1
78
Under the hood

GRAIL: Generated video, lifted to metric 4D, tracked on SONIC, distilled into an RGB policy

Stage 1: generate
Stage 2: track in physics
Stage 3: distill and deploy
3D scene + cameraobject mesh · Infinigen room character fitted to the G1
Video world modelKling: frame + prompt in, 5 or 10 s of video out
Initial estimateGENMO body · WiLoR hands FoundationPose object
Joint optimization$\mathcal{L}_{kp}, \mathcal{L}_{proj}, \mathcal{L}_{depth}, \mathcal{L}_{cont}, \mathcal{L}_{reg}$ on human + object residuals
G1 reference $\tilde q_t$GMR retarget, kinematic + object trajectory
Reference + object $o_t$10 future frames · BPS shape object deltas, contact forces
Adaptor $\pi_\phi$MLP 512-256-128 → 64-d $\Delta z_t$ + 2 hand open / close
SONIC, frozen$\mathcal{G}(z_t + 0.1\,\Delta z_t)$ → 29 joint targets
Isaac Lab · G143 DoF with fingers + the object
Reward $R_t$ → PPOmotion + object + grasp 64 L40 GPUs · ≈ 30 h
Trackers as teacherspick-up: adaptor + SONIC stairs: scene-aware tracker
RGB policyone per skill · 10 Hz writes SONIC tokens
SONIC decodertokens → joint targets
Simulation → real G1real: OAK-D W camera, desktop RTX 5090
Domain randomizationvisual randomization, camera alignment
1 frame
video
4D HOI
known: camera, depth, mesh, body shape
$\Delta z_t$
targets
state $s_t$
state
update only $\pi_\phi$
distill
tokens
head-camera RGB + proprioception
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The tracking loop: a small adaptor steers a frozen SONICThe reference: a generated video, lifted against the known scenePPO trains only the adaptor: motion, object and grasp rewardsThe trackers teach an RGB policy that writes SONIC tokensWhat it costs, and why SONIC stays frozen
Objective
  • Tracker: $(\Delta z_t,\ a^{\text{hand}}_t) = \pi_\phi(s_t, o_t)$, $\ a^{\text{body}}_t = \mathcal{G}(z_t + \lambda\,\Delta z_t)$, $\ \lambda = 0.1$
  • Reconstruction: $\mathcal{L} = \lambda_{kp}\mathcal{L}_{kp} + \lambda_{proj}\mathcal{L}_{proj} + \lambda_{depth}\mathcal{L}_{depth} + \lambda_{cont}\mathcal{L}_{cont} + \lambda_{reg}\mathcal{L}_{reg}$
  • Reward: $R_t = R^{\text{motion}}_t + R^{\text{reg}}_t + R^{\text{obj}}_t + \mathbf{1}\{C_t\}\,R^{\text{grasp}}_t$, grasp term only during contact
  • $R^{\text{motion}}_t = \sum_i w_i \exp\!\big(-\|\tilde x_{i,t} - x_{i,t}\|^2 / \sigma_i^2\big)$; PPO: $\gamma = 0.99$, $\epsilon = 0.2$
Training recipe
  1. Render a known 3D scene; generate 5 or 10 s of video from that frame and a text prompt
  2. Estimate human and object, drop bad tracks, optimize both jointly, retarget with GMR
  3. PPO on 2,000–4,000 motions per run; for terrain, fine-tune SONIC with an 11 × 11 height map
  4. Distill each tracker into an egocentric RGB policy with domain randomization; deploy zero-shot
≈ 14 minper 5 s clip, on one A100
64 GPUsL40 · ≈ 30 h per tracker run
35,000+physically validated motions (project page)
10 HzRGB policy, desktop RTX 5090
The detail that makes it work

A small adaptor on a frozen SONIC. Without SONIC, body tracking degrades (45.0 % success); SONIC alone tracks the body, not the object (39.7 %); together 81.4 %.

Versus its predecessor

vs VIRAL and DoorMan: the teacher tracks generated motions, not a hand-designed staged reward. vs HDMI and ResMimic: one tracker per task family, with fingers.

StartRootsMoveHandsSystem 0SeeSystem 1
79
Lab table · inset: the policy’s camera view
Office kitchen · a crown off a cluttered table
the same recipe,
on stairs?
Indoors (2×) · a person holds a handle
Outdoors · a person holds a handle
Lab stairs, up and down (2×) · nobody holds it
outside the lab,
in one take?
One take: stairs, ground pick-up, trash bin (3×)
CoRL 2026 oral · new demos

Pick-up. Trained on 100 % GRAIL data, deployed zero-shot: 80 % success on unseen objects (oral deck, slide 16).

Stairs. The oral deck reports 100 % success on lab stairs (right); the paper reported 90 %. No trial counts.

Outside the lab. Stairs, a ground pick-up and a trash bin in one take. No success rate is reported here.

Generated data only, deployed on the real robot

80 %
pick-up, unseen objectsoral deck, slide 16 · paper: also 80 %
100 %
on lab stairsoral deck, slide 17 · no trial count
90 %
stair-climbingpaper v2, Sec. 4.3 · no trial count
Clips: GRAIL CoRL 2026 oral deck, slides 16–18, and project page (research.nvidia.com/labs/dair/grail) · every clip shows a real Unitree G1
StartRootsMoveHandsSystem 0SeeSystem 1
80
Results

Generated motions that hold up in physics

Both success rates are measured in a physics simulator: first the generated human motions, then the robot tracker.
Generated videos · one robot-sized character
88.9 %

physics tracking success of the generated human–object motions. DAViD 24.0 %, HOIDiff 15.8 %, CHOIS 10.5 %.

and on the robot,
one policy per motion?
Simulation · a G1 tracks a generated table pick-up
81.4 %

manipulation success, one task-general tracker, 124 motions. ResMimic 49.2 %, HDMI 48.5 %: one policy per task.

and how much
of it is public?
Simulation · every G1 executes a generated motion
35,000+

physically validated motions in the public dataset; over 20,000 in the paper. ≈ 268K downloads.

Unlocked

Robot data from a prompt: a 3D asset and a video model give physics-checked motions that train real-robot policies.

Still open

Still one RGB policy per skill, and no language. Can one model write the commands for every task, on the robot?

Clips: GRAIL project page and CoRL 2026 oral deck · numbers: Tables 1 and 2 of the paper · 35,000+: project page · ≈ 268K downloads: Hugging Face, 19 Nov 2026 (oral deck: ≈ 250K)
StartRootsMoveHandsSystem 0SeeSystem 1
81
Chapter 6  ·  2026

System 1

Can one model write the commands for every task,
on the robot’s own computer?
GR00T 2 Turbo · whole-body loco-manipulation NVIDIA whitepaper · November 2026
82
NVIDIA whitepaper · November 2026

GR00T 2 Turbo: a world-action
model inside the humanoid

Onboard a Unitree G1
48 of 48 popcorn runs completed
83
How it works

Three layers, three clocks, two computers

Reasoning model · S2off the robot · optionalpicks task and target
GR00T 2 Turbo · S1onboard · Jetson AGX Thorplans 1.6 s · ≈ 1 Hz
SONIC · S0onboard · Jetson Orin NXjoint targets · 50 Hz
Unitree G1head + two wrist camerasjoint PD · 1 kHz
joint
targets
proprioception · 50 Hz
whole-body
tokens
camera images + proprioception
WAM API
onboard · two computers

S0 keeps the body up. SONIC turns one token and the robot’s state into joint targets, 50 times a second, on a Jetson Orin NX. The G1’s joint PD control tracks them at 1 kHz.

S1 writes the tokens. GR00T 2 Turbo, on a Jetson AGX Thor, reads the head and wrist cameras. About once per second it plans the next 1.6 s of whole-body motion, as tokens.

S2 picks the task. A reasoning model (a VLM agent, a scripted plan or an operator) chooses the task and marks the target, then calls S1 through the WAM API. It is optional, and off the robot.

Everything else runs onboard. Two computers on the robot and three clocks: a new plan about once per second, joint targets at 50 Hz, joint PD at 1 kHz.

S0 at work: one foot wipes, the other balances (real G1)
who writes
the tokens?
S1 at work: table clean-up into the bins (real G1)
and who picks
the task?
LANGUAGE PROMPT “Clear the table into the bins.” TARGET MARK WAM API GR00T 2 Turbo S1
Schematic: S2 sends a prompt and a target mark to S1
NVIDIA. GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation. Whitepaper, November 2026 · stack: Fig. 9, App. H · clips: project page, speed not stated
StartRootsMoveHandsSystem 0SeeSystem 1
84
Onboard

Fast enough to run on the robot

Three real G1s at once: floor pick-up, popcorn, laundry hamper
Model2B, keptsame as GR00T 2
Horizonunchanged80 actions
ComputerAGX Thoron the G1
BeforeGR00T 2 · workstation
153 msper chunk · RTX 5090
eager449 ms optimized153 ms GR00T 2 · Dexter (dexterous hands)
Compress videonew video encoder
16×per side · was 8×
8× · GR00T 2 16× · Turbo 4× smaller latent grid per frame
Optimizesame checkpoint
38 mswas 225 ms · RTX 5090
eager225 ms optimized38 ms Turbo · G1 checkpoint · 5.85× faster
On the robotJetson AGX Thor
124 mswas 720 ms · 120 W mode
720 615 256 254 182 124 ms TensorRT 5.81× faster step by step
The budget. GR00T 2 is served from a workstation GPU. A humanoid has to carry its computer: an edge GPU with far less compute, for a model that denoises video and actions together.
Before. GR00T 2 predicts one action chunk in a median of 153 ms on a GeForce RTX 5090, already optimized: measured on Dexter, a robot with dexterous hands.
Compress the video. Turbo keeps the 2B transformer and the action horizon. Its video encoder (the Wan 2.2 VAE) compresses 16× per side instead of 8×: a 4× smaller latent grid, so far less for the transformer to process.
Optimize inference. Same weights, measured step by step: runtime optimizations, TensorRT (the largest single cut), CUDA graph replay, FP8 mixed precision, and fewer denoising steps (2 video + 4 action → 1 + 2).
124 ms on the robot’s own Jetson, for a plan that is renewed about once per second. The lighter model has a price: with dexterous hands, Turbo succeeds 34.3 % of the time where GR00T 2 reaches 68.6 %.
Numbers: GR00T 2 whitepaper, Sec. 2, Tables 1 and 27, Fig. 19 · clip: GR00T 2 project page, speed not stated · bars and grids are drawn to scale
StartRootsMoveHandsSystem 0SeeSystem 1
85
The interface

S1 writes 80 tokens, S0 plays them back

0 s1 s2 s3 s S0 plays back S1 writesGR00T 2 Turbo one chunk = 80 tokens = 1.6 s  + gripper widths S0 plays backSONIC · 50 Hz one token every 20 ms→ joint targets at 50 Hz overlap ≈ 0.6 s next chunk ≈ 1 s later and again …
Schematic, to scale: one green cell = one token = 20 ms · how overlapping chunks are merged is not stated

S1 writes a chunk. One prediction is 1.6 s of whole-body motion: 80 tokens in SONIC’s token space. The gripper widths come with it, and go around SONIC.

S0 plays it back. Every 20 ms SONIC takes the next token and the current robot state, and returns joint targets.

A new chunk about once per second. Each plan covers 1.6 s, so consecutive chunks overlap by about 0.6 s. Between two plans SONIC runs about 50 control steps on its own.

One token space, three uses. Teleoperation records the operator’s motion as tokens, simulation replays recorded motions through the same controller, and Turbo predicts tokens.

Why tokens. In the SONIC paper, a VLA that predicts tokens averages 68 % success over three real tasks; predicting body poses instead, 27 %. The token is the contract. So what is inside S1?

Real G1: a deep bend for the cup, played back by SONIC
who else writes
these tokens?
Teleoperationoperator’s motion → tokens Simulationrecorded motions, replayed DeploymentGR00T 2 Turbo → tokens SONICS0 one controller,sim and real
Schematic: one token space, used three times
why tokens,
and not body poses?
68 %a VLA that predicts tokensGR00T N1.5 on SONIC · 3 real tasks
27 %the same VLA predicting body posesSMPL poses instead of tokens
SONIC paper, Table 1B · mean of three real tasks
Chunk and rates: GR00T 2 whitepaper, Fig. 9 and App. H · 68 % vs 27 %: SONIC (Science Robotics 2026), Table 1B · clip: GR00T 2 project page, speed not stated
StartRootsMoveHandsSystem 0SeeSystem 1
86
Under the hood

GR00T 2 Turbo: One 2B transformer denoises future video and 80 tokens together

GR00T 2 pretraining402,730 h human video · 23,500 h robot (G1: 275 h)
Flow matchingone loss on video latents, one on the action chunk
G1 demonstrationsteleoperated as SONIC tokens, plus simulated replays
Camera historyhead + 2 wrist views · 5 frames each, 224 × 320
Wan 2.2 video VAE16× per side (GR00T 2: 8×) → 4× smaller latent grid
GR00T 2 Turbo · S12B video–action transformer one prediction ≈ every 1 s
SONIC · S0one token per 20 ms → joint targets at 50 Hz
Unitree G1joint PD at 1 kHz · head + 2 wrist cameras
Prompt from S2language + target mark, through the WAM API
Language encoderCosmos-Reason1, frozen · read by cross-attention
Future framesof every camera, predicted together with the actions
latents
80 tokens
targets
camera images
text features
joint positions
gripper widths
pretrain
post-train
update
policylearned networkfrozen / not learneddata · signalloss · rewardsimulator · robotactiontraining signal
The control loop: images in, 80 tokens out, 50 Hz joint targetsThree inputs: camera video, the prompt, joint positionsOne prediction denoises future frames and the action chunk togetherTrained by imitation only: flow matching on video and actionsWhat it costs, and what makes it fit on the robot
Objective
  • One input sequence: video latents, state, noisy actions; text by cross-attention
  • Run time: 1 video + 2 action denoising steps; the actions attend to the future frames
  • $\mathcal{L} = \lambda_v\,\mathcal{L}_{\text{video}} + \lambda_a\,\mathcal{L}_{\text{action}}$, flow matching on future video latents and on actions
  • $\mathcal{L}_{\text{action}}$ on $80 \times$ (SONIC token + gripper widths) $= 1.6$ s at 50 Hz; imitation only, no reward
Training recipe
  1. Keep GR00T 2’s transformer (built on Cosmos-Predict2.5-2B); swap its VAE for Wan 2.2
  2. Pretraining (GR00T 2 recipe), three stages: video, video + action, robot-upweighted
  3. Record G1 demonstrations by whole-body teleoperation, directly as SONIC tokens
  4. Post-train per task family: popcorn and spill single-task, pick-and-place multi-task
2Bparameters, the same as GR00T 2
124 msper prediction on the Jetson AGX Thor · was 720 ms
17 hteleoperation for popcorn · 1,500 episodes
1,000 hsimulated replays · 250,000 episodes
The detail that makes it work

Video is redundant. A 16× VAE leaves a 4× smaller latent grid for the same 2B transformer; inference optimization then cuts 720 ms to 124 ms.

Versus its predecessor

vs GR00T 1.7 (VLA), same SONIC and data: 80.8 % vs 62.5 % average success. vs GR00T 2: 16× video compression instead of 8×; dexterous-hand success halves.

StartRootsMoveHandsSystem 0SeeSystem 1
87
Data

Replay real motions in simulation, 250,000 times

Teleoperation is costly. Keep each recorded motion and search for simulated scenes where it still succeeds.
1 · Teleoperate a pickkeep only the recorded whole-body motion
2 · Fit a scene to itSONIC replays it; CMA-ES places table and object
3 · Check the physicslift, then hold under a pull equal to the weight
4 · Render it many timesgenerated rooms, random lighting and camera
Recipe · whitepaper Fig. 59, redrawn as a list
Simulation · Isaac Lab: the G1 lifts the crown
the same task,
on the real table?
Real G1 · the same task on the real table
250,000

simulated episodes, 1,000 hours, in the same format as the real demonstrations.

6.7 → 63.3 %

Out of distribution (unseen scenes), real data only → real + simulated. GR00T 1.7 (VLA): 6.7 → 40.0 %.

73.3 → 83.3 %

In distribution, where the real data were collected. GR00T 1.7: 46.7 → 66.7 %. π0.5: 40.0 → 16.7 %.

Tabletop pick-and-place on the real G1, 30 trials per cell. Out of distribution, with simulated data: 19 successes for GR00T 2 Turbo, 12 for GR00T 1.7. The whitepaper calls that gap suggestive rather than conclusive (p ≈ 0.12).
Clips: GR00T 2 project page (research.nvidia.com/labs/gear/gr00t-2), speed not stated · numbers: whitepaper Table 26 and App. H.3 (pp. 111–112) · the recipe list is redrawn from Fig. 59
StartRootsMoveHandsSystem 0SeeSystem 1
88
Results

Four real tasks on one humanoid

Popcorn: scoop, fill, into the tray (1×)
48 of 48

popcorn runs completed, two full cups in each runGR00T 1.7, same data: 83.3 %

… a table of
six objects?
Table clean-up: can from table to bin
93.3 %

table clean-up, 28 of 30: six objects into the binsGR00T 1.7, same data: 73.3 %

… objects
on the floor?
Ground pick-up: bend, grasp, stand up
83.3 %

ground pick-up, 25 of 30 trials with objects on the floorGR00T 1.7, same data: 50.0 %

… and a spill,
wiped with a foot?
Spill clean-up: wipe with a foot, kneel
46.7 %

spill clean-up, 14 of 30 full sequences completedGR00T 1.7, same data: 43.3 %

80.8 %
GR00T 2 Turboaverage, four tasks
62.5 %
GR00T 1.7VLA · same data
18.5 %
DPdiffusion policy
7.5 %
π0.5VLA · same data

Unlocked

A world-action model inside the humanoid: it plans 1.6 s about once per second; SONIC executes the plan at 50 Hz.

Still open

The token carries motion, not contact force, and the hardest whole-body task succeeds 46.7 %. What comes next?

Clips: real Unitree G1, GR00T 2 project page, cropped; speed stated only for popcorn (1×) · success rates: whitepaper Table 25, App. H.2 (p. 111) · 48 popcorn trials, 30 per other task
StartRootsMoveHandsSystem 0SeeSystem 1
89
Open source

The stack, as repositories

One row per layer: what each repository ships, and its GitHub stars on 19 November 2026.
HOMIE · operator in the cockpit, the real G1 follows
SONIC · released controller in MuJoCo simulation, from the docs
DoorMan · generated door scenes, renders from the README
Gallant · real G1 up and down the stairs (project page)
InternUtopia · README teaser (still image)
Listed11repositories
Stars8,285our sum · 19 Nov
To comeGR00T 2code: December
Hardwarecockpit and glove
OpenHomie626HOMIE cockpit codehardware files on request
TAG43glove CAD, circuit boards,firmware, control code
System 0whole-body control
OpenHomieHOMIE’s leg policy:training code
v2
GR00T-WholeBodyControl3,933Decoupled WBC: 2 lower-body policies, teleoperationSONIC: training code, C++ inference, 3 checkpoints
Writerspolicies from pixels
GR00T-VisualSim2Real447main: VIRAL · teacher + student trainingdoorman: DoorMan · plus a door generator
Isaac-GR00T8,399context: the GR00T N1.x VLAabove System 0 · not counted
Specialistsone skill each
Gallant113voxel-grid locomotion:training code
HoST647standing up: trainingand evaluation code
RoboDuet191 + 39quadruped + arm: training,deployment (2 repositories)
Sim and dataplatforms, dataset
InternUtopia1,309ex GRUtopia: simulationplatform, 6 releases
GRAIL620pipeline, checkpoints;dataset: 267,769 downloads
DiffTactile317differentiable tactilesimulator
The person’s side. HOMIE’s cockpit code is public; the exoskeleton, glove and pedal files come on request. TAG’s tactile glove is open hardware: CAD, circuit boards, firmware and control code in one repository.
System 0. HOMIE’s repository points to a “homie v2” policy inside GR00T-WholeBodyControl: the Decoupled WBC. The same repository holds SONIC, with training code, C++ inference, three checkpoints and VR teleoperation.
Policies from pixels. VIRAL and DoorMan share one repository, one branch each: teacher training, student training, evaluation. Isaac-GR00T, the VLA above System 0, is context: it is not counted in any sum here.
One specialist per skill. Gallant’s voxel-grid locomotion and HoST’s stand-up controller for the humanoid, RoboDuet for a quadruped with an arm. Each ships its training code; RoboDuet also its deployment code.
Simulation and data. GRUtopia lives on as InternUtopia, six releases so far. GRAIL ships its pipeline, tracking checkpoints and a dataset downloaded 267,769 times. DiffTactile is a simulator package.
The eleven repositories above: 8,285 stars, our sum from GitHub on 19 November 2026, Isaac-GR00T left out. Not public yet: the code of the world-action model.
Stars: GitHub API, 19 Nov 2026 · downloads: Hugging Face API, same day · contents: the READMEs · “v2”: OpenHomie README · sums are ours · media: project pages, repository docs and READMEs
StartRootsMoveHandsSystem 0SeeSystem 1
90
Releases

Released, and coming

Public releases, in the order they came out. Amber is announced, not yet released.
SONIC · VR whole-body teleoperation, real G1
GRAIL · simulated G1s executing generated motions
GR00T 2 · page banner: a robot hand turns 1 into 2
SONICFeb 2026open-sourced
GRAILJun 2026code released
GR00T 2Decembercode, announced
Feb 2025
OpenHomieHOMIE: training and deployment code
Nov 2025
Decoupled WBCGR00T-WholeBodyControl opens, for GR00T N1.5 and N1.6
Feb 2026
SONICcheckpoints, C++ inference, VR teleoperation
Mar 2026
Gallanttraining code
Apr 2026
VisualSim2RealVIRAL and DoorMan: training code
Apr 2026
SONIC training codetrain from scratch, or fine-tune
May 2026
GR00T N1.7 + SONICone workflow: collect, fine-tune, deploy
Jun 2026
GRAILcode · checkpoints in July · dataset on Hugging Face
Nov 2026
GR00T 2whitepaper and project page
December
GR00T 2 codethe page’s button: “Code — Available in December”
Controllers first. HOMIE’s code came out in February 2025, the month of its paper. Its second version, the Decoupled WBC, opened GR00T-WholeBodyControl that November; SONIC followed in February 2026 with checkpoints, C++ inference and VR teleoperation.
Then the training code. Gallant in March, SONIC and the repository of VIRAL and DoorMan in April. In May, SONIC and Isaac-GR00T meet in one workflow: collect data with SONIC, fine-tune GR00T N1.7, deploy on the G1.
Then the data. GRAIL’s pipeline in June, its tracking checkpoints in July. Its dataset on Hugging Face, extended twice in November 2026, holds over 35,000 physically validated motions.
Now the world-action model. The GR00T 2 whitepaper and its project page are dated November 2026. The code button on that page is not active yet; it reads “Available in December”.
More releases are on the way; this talk stops at what is public.
Dates: each repository’s README news; first commit for OpenHomie, Gallant and VisualSim2Real · GR00T 2: project page · the list is not to scale · clips: SONIC, GRAIL and GR00T 2 project pages
StartRootsMoveHandsSystem 0SeeSystem 1
91
Recap · Roots, Move, Hands

The stack as a relay

Each work’s limit is the next work’s title.
RootsBefore the humanoid: can a robot touch, walk and work?2023 – 2024
2024DiffTactileTouch you can simulate and differentiate
2024RoboDuetTwo policies make legs and arm one body
2024GRUtopiaA simulated city for agents on legged robots
MoveCan a humanoid’s legs go anywhere, and get back up?2025
2025HoSTStand up from many postures, no motion data
2025BeamDojoAim every footstep: stepping stones and beams
2025VB-ComA blind policy takes over when vision fails
2025GallantOne LiDAR voxel policy: floor, walls, ceiling
HandsWho drives the upper body, and where do demonstrations come from?2025 – 2026
2025HOMIEA $500 cockpit: one person drives the humanoid
2026FORGEOne force-aware policy for legs and arms
2026TAGThe operator feels the robot’s fingertip contact
2026OPFAOne action latent for grippers and hands
The baton goes on to System 0, See and System 1.
Card years: first public version (arXiv) · venues and full citations: the reference slides
StartRootsMoveHandsSystem 0SeeSystem 1
92
Recap · System 0, See, System 1

The stack as a relay, continued

Each work’s limit is the next work’s title.
System 0Can one controller execute any whole-body motion, whoever writes the command?2025 – 2026
2025Decoupled WBCRL legs, IK arms, commanded by a VLA
2025SONICOne tracker for almost any motion, one token
SeeCan a policy write the commands from the robot’s own camera, trained only in simulation?2025 – 2026
2025VIRALPixels to actions, trained only in simulation
2025DoorManAn RGB policy opens doors it never saw
2026GRAILRobot data generated by a video world model
System 1Can one model write the commands for every task, on the robot’s own computer?2026
2026GR00T 2 TurboA world-action model onboard writes the tokens
The stackThe three roads meet on one Unitree G12026
S0SONICTurns each token into joint targets, at 50 Hz
DATASimulated replays250,000 episodes from real G1 motions
S1GR00T 2 TurboWrites 80 tokens about once per second
No person writes the commands in this stack. So what replaced the cockpit?
The stack: GR00T 2 whitepaper (NVIDIA, Nov. 2026), Fig. 9 and App. H · card years: first public version (arXiv, repository or whitepaper) · venues: the reference slides
StartRootsMoveHandsSystem 0SeeSystem 1
93
Takeaways

Three things building the stack taught us

01

The controller
became an API

velocitygoal pointjoint anglestokens anywriter S0SONIC 64 numbers

The command changed with each controller: a velocity, a goal point, joint angles. SONIC ends it with 64 numbers that any layer above can write.

RoboDuet  Gallant  HOMIE  SONIC
VIRAL · increments on HOMIE’s controller
02

The writer changed
three times

a persona sim-trained policya VLAa world-action model command

First a person in a cockpit. Then policies trained in simulation, and a VLA. Today a world-action model on the robot writes the tokens.

HOMIE  VIRAL · DoorMan · GRAIL
  GR00T N1.5 on SONIC  GR00T 2 Turbo
03

Data stopped needing
a person per episode

rewardsteleop≈ 700 h35,000+250,000 mocapmotionsepisodes HoSTHOMIESONICGRAILGR00T 2

Hand-written rewards, then teleoperation. Then ≈ 700 h of motion capture, generated video, and 250,000 simulated replays of real motions.

HOMIE  SONIC  GRAIL  GR00T 2 Turbo
teleoperation still seeds the replays
No single model replaced the person in the cockpit: a stack did, one layer and one interface at a time.
64 numbers, ≈ 700 h: SONIC (Science Robotics 2026) · 35,000+ motions: GRAIL project page · 250,000 episodes: GR00T 2 whitepaper, App. H · the three diagrams are schematics
StartRootsMoveHandsSystem 0SeeSystem 1
94
Outlook

What is still open

Contact

Force, not only motion

The token from S1 to S0 carries motion. Contact force is not represented explicitly, and S1 takes no touch input. Tactile gloves and force estimates are a start.

Seed: TAG · blindfolded operator feels each press
Also: FORGE (wrist force, no sensor) · DiffTactile
Perception

Eyes inside System 0

SONIC is blind: it reads proprioception and the command. Terrain reaches it only through the layer above. A general tracker with its own eyes is still missing.

Seed: Gallant · a LiDAR policy ducks under a panel
Also: GRAIL · height map into SONIC, in simulation
Horizon

Long tasks, few mistakes

Spill clean-up, the hardest whole-body task, succeeds in 14 of 30 trials (46.7 %). S1 sees 0.8 s of history; longer plans are left to a slower reasoning layer.

Seed: GR00T 2 Turbo · trash to a cafeteria bin
Also: S2 · a reasoning model picks task and target
If the last four years asked “who writes the command?”, the next ask “what can each layer feel, see and remember?”
Clips: TAG, Gallant and GR00T 2 project pages, real robots (playback speeds not stated there) · limits: GR00T 2 whitepaper, pp. 26–28 and Table 25; SONIC, Science Robotics 2026
StartRootsMoveHandsSystem 0SeeSystem 1
95
References · Roots, Move, Hands

Works in this talk

Roots
DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation. ICLR 2024
RoboDuet: Learning a Cooperative Policy for Whole-body Legged Loco-Manipulation. IEEE Robotics and Automation Letters 10(5), 2025
GRUtopia: Dream General Robots in a City at Scale. Technical report, arXiv:2407.10943, 2024
Move
Learning Humanoid Standing-up Control across Diverse Postures (HoST). RSS 2025 · Outstanding Systems Paper Award finalist
BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds. RSS 2025
Move continued
VB-Com: Learning Vision-Blind Composite Humanoid Locomotion Against Deficient Perception. ICRA 2026
Gallant: Voxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains. CVPR 2026 · Highlight
Hands
HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit. RSS 2025
FORGE: Force-Aware Multi-Granularity Control for Efficient Humanoid Loco-Manipulation. arXiv:2611.17142, 2026
Feel Robot Feels: Tactile Feedback Array Glove for Dexterous Manipulation (TAG). arXiv:2603.28542, 2026 · in submission
One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation (OPFA). ICRA 2026
HoST: RSS lists the award as “Outstanding Systems Paper Award” finalist.
StartRootsMoveHandsSystem 0SeeSystem 1
96
References · System 0, See, System 1

Works in this talk, continued

System 0
GR00T-WholeBodyControl: the Decoupled WBC and SONIC controllers. github.com/NVlabs/GR00T-WholeBodyControl · open source since Nov. 2025
SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control. Science Robotics 11(117), 2026 · arXiv:2511.07820, 2025
See
VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation. CVPR 2026
See continued
Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer (DoorMan). CVPR 2026
GRAIL: Generating Humanoid Loco-Manipulation from World Models. CoRL 2026 oral · Best Paper Award Finalist
System 1
Isaac-GR00T: the GR00T N1 to N1.7 vision-language-action models. github.com/NVIDIA/Isaac-GR00T · open source since Mar. 2025
GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation. NVIDIA whitepaper, November 2026
Also co-authored: SMAP: Self-supervised Motion Adaptation for Physically Plausible Humanoid Whole-body Control. arXiv:2505.19463, 2025
TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperation. arXiv:2505.12748, 2025
Also named on the slides, as baselines, predecessors or components of the works above: Deep Whole-Body Control · GAMMA · GeFF · Walk These Ways · PIM · OpenTelevision · AnyTeleop · Telekinesis · Mobile-TeleVision · HumanPlus · OmniH2O · Seer · TWIST2 · Manus glove · BeyondMimic · Any2Track · DAViD · HOIDiff · CHOIS · ResMimic · HDMI · π0.5 · Diffusion Policy (DP). The layer names System 2 / 1 / 0 follow the GR00T 2 whitepaper. All clips and figures come from the authors’ public pages, videos and papers.
StartRootsMoveHandsSystem 0SeeSystem 1
97
Thank you
From a robot dog with an arm
to one stack for the whole body.
Questions?
www.qingweiben.com  ·  elgceben@gmail.com