Isaac Teleop Setup (CloudXR / DeviceIO, in-process)

Isaac Teleop Setup (CloudXR / DeviceIO, in-process)#

This page documents the Isaac Teleop / CloudXR bring-up for G1 with a Thor backpack that drives GR00T-WholeBodyControl directly from the headset — no separate publisher container, no host ~/.cloudxr sharing. The CloudXR runtime is hosted in-process by pico_manager_thread_server.py --input-source isaac-teleop via the isaacteleop[cloudxr] Python package.

Scope

This workflow is currently documented and supported only for G1 + Thor backpack.

Prerequisites#

  1. Completed the Quick Start — you can run the sim2sim loop (includes installing the deployment and downloading model checkpoints).

  2. Completed the VR Teleop Setup — .venv_teleop is ready and install_pico.sh has been run on Thor.

This page is a condensed, repo-specific version of the upstream Isaac Teleop docs:

Step 1: Prepare the Thor Host#

Install the prerequisites on Thor (skip any you’ve already done for the rest of the deploy):

sudo apt install -y build-essential curl git-lfs

ARCH=$(uname -m)
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-${ARCH}.sh"
bash "Miniforge3-Linux-${ARCH}.sh" -b -p "$HOME/miniforge3"
"$HOME/miniforge3/bin/conda" init
source ~/.bashrc

git lfs install

For Thor performance, enable max power mode before teleoperation:

sudo nvpmodel -m 0
sudo jetson_clocks

Optional thermal / over-current check:

cat /sys/class/hwmon/hwmon*/oc*_event_cnt

Step 2: Install isaacteleop[cloudxr]#

install_pico.sh (run from the VR Teleop Setup Step 3) installs the isaacteleop[cloudxr] package into .venv_teleop from the public NVIDIA index:

# Already wired into install_pico.sh; shown here for reference
uv pip install 'isaacteleop[cloudxr]~=1.3.0' --prerelease=allow \
    --extra-index-url https://pypi.nvidia.com

It also seeds ~/cloudxr.env with NV_DEVICE_PROFILE=Quest3 (override by editing the file). CloudXRLauncher reads this on startup.

Step 3: Connect the XR Client#

The streamer launches the CloudXR runtime as a subprocess of the Python loop the first time you start it (Step 4). Until you connect the headset, IsaacTeleopClient.start_streaming() will retry quietly with "no XR session yet".

  • Open the Isaac Teleop Web Client in the headset browser

  • Enter the IP address of the Thor host

  • Accept the self-signed certificate at https://<thor-ip>:48322

  • Return to the client page and click Connect

For quick validation, the same client URL can also be opened in a desktop browser.

If you prefer to run the WebXR client from source instead of the hosted client, follow the CloudXR/WebXR build instructions linked from the Isaac Teleop Quick Start.

Important

The streamer will fail to acquire OpenXR until the XR client is connected. Either connect the headset first, or start the streamer and watch for the Isaac Teleop session initialized. log line once you do.

Step 4: Start the C++ Deployment#

From gear_sonic_deploy/:

export TensorRT_ROOT=$HOME/TensorRT   # only if not already in ~/.bashrc
./docker/run-ros2-dev.sh

# inside the container (setup_env.sh is sourced automatically):
just build                            # first run only
./deploy.sh --input-type zmq_manager real
# Wait until you see "Init done"

Step 5: Launch the Teleop Streamer#

From the repo root:

source .venv_teleop/bin/activate
python gear_sonic/scripts/pico_manager_thread_server.py --manager \
    --input-source isaac-teleop

# If running offboard with a display, add visualization:
#   --vis_vr3pt --vis_smpl

Watch the streamer logs for Isaac Teleop session initialized. — that means CloudXR + DeviceIO are both up. The streamer then prints Manager controls: A+X=toggle mode, A+B+X+Y=start/stop policy once the headset is connected and body data starts flowing.

For controls and calibration, see VR Whole-Body Teleop.

Step 6: Start Camera Streaming#

Stream cameras to the headset via upstream IsaacTeleop’s camera_streamer.sh. If you don’t have the IsaacTeleop repo yet, clone it first:

git clone --recurse-submodules https://github.com/NVIDIA/IsaacTeleop.git

Then build and launch the camera streamer:

cd IsaacTeleop
cd examples/camera_streamer

./camera_streamer.sh build
./camera_streamer.sh list-cameras

Run with local cameras streaming to the XR headset:

source scripts/setup_cloudxr_env.sh
./camera_streamer.sh run --source local --mode xr

# Optionally specify a camera configuration:
#   --config config/single_camera.yaml
#   --config config/multi_camera.yaml

The --input-source isaac-teleop flag also works with Data Collection.

Troubleshooting#

RuntimeError: Failed to get OpenXR system: -35#

In this setup, that error usually means the XR client is not connected yet. Re-check:

  • The headset / web client is fully connected to https://<thor-ip>:48322

  • The CloudXR runtime subprocess is still alive (look for the Isaac Teleop session initialized. log)

  • ~/cloudxr.env exists and has the right NV_DEVICE_PROFILE for your headset

gear_sonic_deploy build error#

If just build fails, rebuild from scratch inside the container (Step 4):

rm -rf build
just build

isaacteleop import error#

Re-run install_pico.sh to reinstall isaacteleop[cloudxr]~=1.3.0 into .venv_teleop. If pypi.nvidia.com is unreachable, check your network and the --extra-index-url flag.

Body data not arriving#

The streamer logs [IsaacTeleopReader] No DeviceIO data for 5.0s, flagging disconnect if the headset stops feeding body data. Confirm:

  1. The headset is still connected to CloudXR (Step 3).

  2. The Pico body trackers are paired and calibrated (see VR Teleop Setup → Motion Tracker Setup).

  3. The first time the schema runs, watch for [IsaacTeleopReader] Unrecognised body_data schema: type=... — if you see it, the upstream FullBodyTrackerPico.get_body_pose().data shape changed and _body_data_to_24x7() in gear_sonic/utils/teleop/input_readers.py needs an extra branch for the new layout.

Contents