curobo.examples.reference.live_volumetric_mapping_mpc module

Fuse live RGB-D frames into a cuRobo volumetric map and MPC scene.

This reference example uses an Intel RealSense backend for live RGB-D capture: it aligns depth to the RGB stream, converts each frame into CameraObservation, and integrates the observations into Mapper. The script is intended as a live mapping and control demo: it updates the TSDF continuously, opens a Viser preview of the reconstruction, recomputes an ESDF slice at the viewer refresh cadence, and runs a Franka MPC loop with a draggable target against the live ESDF collision grid. It does not write rendered PNGs or meshes.

Live volumetric mapping MPC target control

The dynamic-obstacle clip shows the robot reacting to obstacles introduced into the live depth scene.

Live MPC reacting to dynamic obstacles

Install the RealSense Python bindings before running the example:

uv pip install pyrealsense2

Run live fixed-camera mapping until Ctrl-C. The example opens an interactive Viser point-cloud preview while integrating:

python -m curobo.examples.reference.live_volumetric_mapping_mpc

The current RealSense backend provides RGB-D frames and intrinsics, but it does not provide a camera-to-world pose. This example applies fixed -90 degree X-axis and +90 degree Y-axis camera-frame rotations, then uses the first valid depth frame to translate the camera so that the initial point cloud mean lands in front of the robot base. Use --camera-pose x y z qw qx qy qz to set a base transform before this fixed rotation and first-frame offset.

class RealSenseRGBDStream(
width,
height,
fps,
device,
rs_module=None,
)

Bases: object

RealSense RGB-D stream aligned to the color camera.

Parameters:
__init__(
width,
height,
fps,
device,
rs_module=None,
)
Parameters:
stop()

Stop the RealSense pipeline.

Return type:

None

warmup(num_frames)

Wait for initial frames so RealSense auto-exposure can settle.

Return type:

None

Parameters:

num_frames (int)

wait_for_observation(
pose,
timeout_ms=5000,
)

Read one aligned RGB-D frame and convert it to CameraObservation.

Return type:

Optional[CameraObservation]

Parameters:
class LiveEsdfMpc(
voxel_grid,
device,
visualizer,
)

Bases: object

MPC controller with a draggable target and live ESDF collision.

Parameters:

  • voxel_grid (VoxelGrid)

  • device (str)

  • visualizer (ViserVisualizer)

__init__(
voxel_grid,
device,
visualizer,
)
Parameters:
step()

Advance MPC by one action using the aliased live ESDF tensor.

main()
Return type:

None