curobo.perception module

Perception utilities module.

This module provides sensor processing utilities for robot perception tasks, including robot segmentation and volumetric mapping from depth images.

Example (robot segmentation):
from curobo.perception import RobotSegmenter
from curobo.types import CameraObservation, JointState

# Create segmenter
segmenter = RobotSegmenter.from_robot_file(
    robot_file="franka.yml",
    distance_threshold=0.05,
)

# Segment robot from depth image
camera_obs = CameraObservation(
    depth_image=depth_tensor,
    intrinsics=camera_intrinsics,
    pose=camera_pose,
)
joint_state = JointState(position=current_joint_angles)

mask, filtered_depth = segmenter.get_robot_mask(camera_obs, joint_state)
# mask: binary mask of robot pixels
# filtered_depth: depth image with robot removed
Example (volumetric mapping):
from curobo.perception import FilterDepth, Mapper, MapperCfg

mapper = Mapper(MapperCfg(extent_meters_xyz=(2.0, 2.0, 2.0), voxel_size=0.02))
depth_filter = FilterDepth(image_shape=depth.shape[-2:])
filtered, valid = depth_filter(depth.unsqueeze(0))
camera_obs.depth_image = filtered[0]
mapper.integrate(camera_obs)
Example (robot pose estimation):
from curobo.perception import DetectorCfg, PoseDetector, RobotMesh

mesh = RobotMesh.from_kinematics(kinematics)
detector = PoseDetector(mesh, DetectorCfg())
result = detector.detect_from_points(pointcloud, config=joint_state.position)
pose = result.pose
class DetectorCfg(
n_mesh_points_coarse=500,
n_observed_points_coarse=2000,
n_rotation_samples=64,
n_iterations_coarse=50,
distance_threshold_coarse=0.5,
n_mesh_points_fine=2000,
n_observed_points_fine=10000,
n_iterations_fine=50,
distance_threshold_fine=0.01,
use_svd=False,
use_huber_loss=True,
huber_delta=0.02,
save_iterations=False,
device_cfg=<factory>,
)

Bases: object

Configuration for pose detector (point-to-plane ICP with Huber loss).

Parameters:
n_mesh_points_coarse: int = 500
n_observed_points_coarse: int = 2000
n_rotation_samples: int = 64
n_iterations_coarse: int = 50
distance_threshold_coarse: float = 0.5
n_mesh_points_fine: int = 2000
n_observed_points_fine: int = 10000
n_iterations_fine: int = 50
distance_threshold_fine: float = 0.01
use_svd: bool = False
use_huber_loss: bool = True
huber_delta: float = 0.02
save_iterations: bool = False
device_cfg: curobo._src.types.device_cfg.DeviceCfg
__init__(
n_mesh_points_coarse=500,
n_observed_points_coarse=2000,
n_rotation_samples=64,
n_iterations_coarse=50,
distance_threshold_coarse=0.5,
n_mesh_points_fine=2000,
n_observed_points_fine=10000,
n_iterations_fine=50,
distance_threshold_fine=0.01,
use_svd=False,
use_huber_loss=True,
huber_delta=0.02,
save_iterations=False,
device_cfg=<factory>,
)
Parameters:
Return type:

None

class FilterDepth(
image_shape,
depth_minimum_distance=0.1,
depth_maximum_distance=10.0,
flying_pixel_threshold=0.5,
bilateral_kernel_size=5,
bilateral_sigma_spatial=10.0,
bilateral_sigma_depth=0.1,
device='cuda',
num_batch=1,
)

Bases: object

GPU-accelerated depth filtering using fused Warp kernels.

Combines range filtering, flying pixel detection, and bilateral smoothing into a single efficient GPU pass. Pre-allocates buffers at initialization for zero-allocation filtering during runtime.

Operates strictly on batched depth tensors of shape (B, H, W). Set num_batch > 1 at init to pre-allocate batched output buffers so all images in the batch are filtered in a single kernel launch. Single-image callers must explicitly pass depth.unsqueeze(0) and index filtered[0] on return.

Parameters:

  • image_shape (Tuple[int, int]) – Tuple of (height, width) for each depth image.

  • depth_minimum_distance (float) – Minimum valid depth in meters. Default: 0.1m.

  • depth_maximum_distance (float) – Maximum valid depth in meters. Default: 10.0m.

  • flying_pixel_threshold (Optional[float]) – Filter aggressiveness 0.0-1.0 for flying pixels. Higher = more aggressive. Set to None to disable. Default: 0.5.

  • bilateral_kernel_size (Optional[int]) – Size of bilateral kernel (must be odd). Set to None to disable bilateral filtering. Default: 5.

  • bilateral_sigma_spatial (float) – Spatial sigma in pixels. Default: 2.0.

  • bilateral_sigma_depth (float) – Depth sigma in meters. Default: 0.05.

  • device (str) – CUDA device. Default: “cuda”.

  • num_batch (int) – Pre-allocate buffers for this batch size. Default: 1.

Initialize FilterDepth with configuration and pre-allocated buffers.

__init__(
image_shape,
depth_minimum_distance=0.1,
depth_maximum_distance=10.0,
flying_pixel_threshold=0.5,
bilateral_kernel_size=5,
bilateral_sigma_spatial=10.0,
bilateral_sigma_depth=0.1,
device='cuda',
num_batch=1,
)

Initialize FilterDepth with configuration and pre-allocated buffers.

Parameters:

  • image_shape (Tuple[int, int])

  • depth_minimum_distance (float)

  • depth_maximum_distance (float)

  • flying_pixel_threshold (float | None)

  • bilateral_kernel_size (int | None)

  • bilateral_sigma_spatial (float)

  • bilateral_sigma_depth (float)

  • device (str)

  • num_batch (int)

update_config(
depth_minimum_distance=None,
depth_maximum_distance=None,
flying_pixel_threshold=None,
bilateral_sigma_depth=None,
)

Update filter configuration without reallocating buffers.

Only updates the specified parameters. Set to None to keep current value.

Parameters:

  • depth_minimum_distance (float | None) – New minimum depth.

  • depth_maximum_distance (float | None) – New maximum depth.

  • flying_pixel_threshold (float | None) – New flying pixel threshold (set to 0 to disable).

  • bilateral_sigma_depth (float | None) – New bilateral depth sigma.

classmethod from_config(
config,
image_shape,
device='cuda',
num_batch=1,
)

Create FilterDepth from a configuration object.

Parameters:

  • config (curobo._src.perception.filter_depth.FilterDepthConfig) – FilterDepthConfig with filter parameters.

  • image_shape (Tuple[int, int]) – Tuple of (height, width) for depth images.

  • device (str) – CUDA device.

  • num_batch (int) – Pre-allocate buffers for this batch size.

Returns:

Configured FilterDepth instance.

Return type:

curobo._src.perception.filter_depth.FilterDepth

class Mapper(config)

Bases: object

Volumetric mapper using block-sparse storage.

Encapsulates block-sparse TSDF integrator with ESDF generation. Provides a simplified API compared to BlockSparseESDFIntegrator.

Parameters:

config (MapperCfg) – MapperCfg with grid and integration parameters.

Example

config = MapperCfg(
    extent_meters_xyz=(2.0, 2.0, 2.0),
    voxel_size=0.005,
)
mapper = Mapper(config)
for obs in observations:

mapper.integrate(obs)

voxel_grid = mapper.compute_esdf() mesh = mapper.extract_mesh()

Initialize Mapper with configuration.

__init__(config)

Initialize Mapper with configuration.

Parameters:

config (curobo._src.perception.mapper.mapper_cfg.MapperCfg)

property integrator: curobo._src.perception.mapper.integrator_esdf.BlockSparseESDFIntegrator

Access underlying integrator for advanced operations.

property tsdf: curobo._src.perception.mapper.storage.BlockSparseTSDF

Access the BlockSparseTSDF storage.

integrate(
*args,
observation=None,
camera_observation=None,
lidar_observation=None,
)

Integrate camera and/or LiDAR observations into the TSDF.

Positional mapper.integrate(obs) calls remain supported. The observation= keyword is retained as a deprecated alias; prefer camera_observation= and/or lidar_observation= for new code. For geometry-only mapping, pass a cached all-zero uint8 RGB image with the same batch and image shape as the depth or range image. When using lower-level configs that expose color resolution, keep color_grid_size=1. This keeps geometry-only and colored mapping on the same path; local benchmarks showed color work is typically less than 10% of integration time.

Parameters:
Return type:

None

clear_region(
bounds_min,
bounds_max,
)

Clear dynamic map contents inside a conservative world-space AABB.

Blocks remain allocated. The cached ESDF voxel grid is invalidated and must be recomputed with compute_esdf.

Return type:

int

clear_blocks(pool_indices)

Clear dynamic map contents for explicit block-pool indices.

Return type:

int

clear_instance_slot(
instance_slot,
)

Clear one retired compact instance slot from mapped categorical state.

Parameters:

instance_slot (int)

Return type:

None

compute_esdf(
esdf_origin=None,
esdf_voxel_size=None,
semantic_filter=None,
)

Compute ESDF from current TSDF and return as VoxelGrid.

Parameters:
Returns:

VoxelGrid containing the signed distance field for collision checking.

Return type:

curobo._src.geom.types.VoxelGrid

extract_mesh(
refine_iterations=0,
surface_only=True,
semantic_filter=None,
)

Extract a triangle-soup mesh using GPU marching cubes.

Parameters:
Returns:

Mesh object with vertices, faces, and colors.

Return type:

curobo._src.geom.types.Mesh

extract_textured_mesh(
texture_observations,
refine_iterations=0,
surface_only=True,
camera_min_distance=None,
camera_max_distance=None,
texture_depth_tolerance_m=None,
semantic_filter=None,
)

Extract a TSDF mesh with visibility-tested RGB texture.

This works with TSDF geometry built from camera or LiDAR observations. Texture observations require RGB images, intrinsics, and poses. Optional depth images must be aligned float32 camera-frame z depth in meters; when omitted, visibility depth is rendered from the current TSDF.

Parameters:
Return type:

curobo._src.geom.types.Mesh

extract_occupied_points(
sample_budget=2000,
*,
instance_slot=None,
semantic_filter=None,
voxel_selection=None,
seed=0,
)

Sample observed surfaces into fixed-capacity CUDA buffers with a GPU count.

sample_budget is the number of proposals before near-zero pruning; rejected points are not refilled. Optional filters intersect: an instance slot, fused semantic features, and uint8 CUDA voxel_selection with shape [max_blocks,block_size**3]. Geometry reads use the unfiltered TSDF. Source voxel indices identify the nearest crossing endpoint. Results are borrowed until the next same-budget query; copy persistent geometry. Warm up the budget before CUDA graph capture. Voxel evidence is unchanged.

Parameters:
Return type:

curobo._src.perception.mapper.extractor_points.OccupiedPoints

extract_occupied_voxels(
surface_only=True,
sdf_threshold=None,
*,
subvoxel_factor=1,
max_points=None,
texture_observations=None,
camera_min_distance=None,
camera_max_distance=None,
texture_depth_tolerance_m=None,
semantic_filter=None,
)

Extract occupied voxel centers and per-voxel block indices.

Parameters:

  • surface_only (bool) – If True, only extract voxels near the TSDF surface. If False, include inside voxels too.

  • sdf_threshold (float | None) – Surface threshold. Defaults to the underlying TSDF integrator’s voxel size.

  • subvoxel_factor (int) – Number of sample points per extracted voxel axis.

  • max_points (int | None) – Optional cap applied after subvoxel expansion by deterministically downsampling source voxels before expansion.

  • texture_observations (curobo._src.types.camera.CameraObservation | collections.abc.Sequence[curobo._src.types.camera.CameraObservation] | None) – Optional RGB observations used to color output samples when they pass visibility-depth checks.

  • camera_min_distance (float | None) – Minimum camera-frame z used for texture projection.

  • camera_max_distance (float | None) – Maximum camera-frame z used for texture projection.

  • texture_depth_tolerance_m (float | None) – Absolute visibility-depth tolerance. If None, each sample uses the mapper default tolerance policy.

  • semantic_filter (curobo._src.perception.mapper.filter_semantic.SemanticFilter | None) – Optional fused-feature filter applied before voxel generation.

Returns:

OccupiedVoxels with voxel centers, per-voxel block pool indices, and a view over per-block storage.

Return type:

curobo._src.perception.mapper.storage.OccupiedVoxels

extract_matching_feature_voxels(
feature_vector,
top_k,
surface_only=False,
sdf_threshold=None,
minimum_score=None,
feature_projector=None,
)

Extract voxels from the top-k blocks most similar (cosine) to feature_vector. Requires MapperCfg.feature_dim > 0.

minimum_score (cosine in the query feature space) drops matched blocks below the cut after the top-k selection, so the voxel-extraction kernel skips work proportional to filtered blocks. Pair a generous top_k with minimum_score to get all matches above a quality threshold.

Returns a MatchedVoxels carrying the extracted voxels plus parallel (K,) block_pool_idx / block_scores tensors in descending score order. block_pool_idx can be passed directly to clear_blocks to drop the matched blocks from the map.

feature_projector optionally maps stored block features into the query space before scoring. This keeps model-specific heads (e.g. RADIO -> SigLIP) outside the mapper while still using the mapper’s all-allocated-block scoring and extraction path.

Parameters:
Return type:

curobo._src.perception.mapper.storage.MatchedVoxels

get_matching_feature_voxels(
feature_vector,
top_k,
surface_only=False,
sdf_threshold=None,
minimum_score=None,
feature_projector=None,
)

Compatibility wrapper for extract_matching_feature_voxels.

Parameters:
Return type:

curobo._src.perception.mapper.storage.MatchedVoxels

reset()

Reset mapper for new scene.

Return type:

None

save_blocks(file_path)

Save active mapper TSDF blocks to a compact checkpoint.

Parameters:

file_path (str | os.PathLike[str])

Return type:

None

classmethod load_blocks(
file_path,
target_cfg,
import_weight=None,
)

Construct a mapper from target_cfg and import block checkpoint data.

Parameters:
Return type:

curobo._src.perception.mapper.mapper.Mapper

import_blocks(
file_path,
import_weight=None,
)

Import active TSDF blocks from disk into this empty mapper.

Parameters:

  • file_path (str | os.PathLike[str]) – Checkpoint written by save_blocks.

  • import_weight (float | None) – Optional constant confidence to assign to imported dynamic TSDF/RGB/feature accumulators. None preserves saved weights.

Returns:

Number of imported active blocks.

Return type:

int

get_stats(
scan_pool=True,
scan_hash=False,
)

Get mapper statistics.

Parameters:

  • scan_pool (bool) – Forwarded to the integrator’s stats call. When True (default) returns the ground-truth active_blocks and the holes invariant via an O(num_allocated) GPU reduction. When False, falls back to the cheap num_allocated - free_count approximation.

  • scan_hash (bool) – Forwarded to the integrator’s stats call. When True, adds hash table occupancy stats via an O(hash_capacity) reduction. Periodic use only.

Returns:

Dictionary with block usage, memory stats, frame count, and last_integration_kernel_timings_ms. See BlockSparseTSDF.get_stats for the full block-pool key list.

Return type:

dict

memory_usage_mb()

Get total GPU memory usage in megabytes.

Return type:

float

update_static_obstacles(
scene,
env_idx=0,
)

Update static obstacles in the TSDF from scene collision tensors.

This method clears the static SDF channel and re-stamps all primitives from the provided scene. The static channel does not decay and is combined with the dynamic (depth) channel using min() for collision.

Note: This is not intended to be called every frame. Use it when static obstacle poses change (e.g., after robot reconfiguration).

Parameters:
Return type:

None

render(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render depth and normals from current TSDF.

Parameters:
Returns:

Tuple of (depth, normals, valid_mask). Unbatched single-camera inputs return (H, W), (H, W, 3), and (H, W); batched inputs return (N, H, W), (N, H, W, 3), and (N, H, W).

Return type:

Tuple[torch.Tensor, torch.Tensor, torch.Tensor]

render_features(
intrinsics,
pose,
image_shape,
image_patch=None,
semantic_filter=None,
)

Raycast compact fused-feature references from the current TSDF.

Parameters:
Returns:

A RenderedFeatures result. Its RenderedFeatures.features method materializes only requested patches and channel slices.

Return type:

curobo._src.perception.mapper.storage.RenderedFeatures

render_color(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render depth, normals, and color from current TSDF.

Parameters:
Returns:

Tuple of (depth, normals, color, valid_mask). Unbatched single-camera inputs keep (H, W) and (H, W, 3) shapes; batched inputs include a leading camera dimension.

Return type:

Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]

render_depth(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render only depth image.

Parameters:
Returns:

(H, W) or (N, H, W) in meters.

Return type:

depth

render_color_only(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render only color image.

Parameters:
Returns:

(H, W, 3) or (N, H, W, 3) uint8 RGB.

Return type:

color

render_shaded(
intrinsics,
pose,
image_shape,
light_direction=(0.0, 0.0, 1.0),
ambient=1.0,
use_color=True,
semantic_filter=None,
)

Render Lambertian-shaded image.

Parameters:
Returns:

(H, W, 3) or (N, H, W, 3) uint8 RGB shaded image.

Return type:

shaded

render_depth_colormap(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render depth as colormap for visualization.

Parameters:
Returns:

(H, W, 3) or (N, H, W, 3) uint8 RGB colormap.

Return type:

colormap

render_normal_colormap(
intrinsics,
pose,
image_shape,
semantic_filter=None,
)

Render normals as colormap for visualization.

Parameters:
Returns:

(H, W, 3) or (N, H, W, 3) uint8 RGB colormap.

Return type:

colormap

class MapperCfg(
extent_meters_xyz,
voxel_size=0.005,
esdf_voxel_size=0.05,
extent_esdf_meters_xyz=None,
grid_center=None,
truncation_distance=0.04,
minimum_tsdf_weight=0.1,
depth_minimum_distance=0.1,
depth_maximum_distance=10.0,
decay_factor=1.0,
frustum_decay_factor=1.0,
block_size=8,
hash_load_factor=0.5,
roughness=3.0,
color_grid_size=1,
seeding_method='gather',
edt_solver='pba',
enable_static=False,
static_obstacle_color=(20, 20, 20),
num_cameras=1,
image_height=None,
image_width=None,
texture_num_cameras=None,
texture_camera_image_height=None,
texture_camera_image_width=None,
lidar_num_sensors=0,
lidar_image_height=None,
lidar_image_width=None,
lidar_feature_grid_height=None,
lidar_feature_grid_width=None,
lidar_linear_interpolation_max_allowable_difference_vox=2.0,
lidar_nearest_interpolation_max_allowable_dist_to_ray_vox=0.5,
max_visible_blocks_per_lidar_integration=None,
max_support_pixels_per_block_lidar=8,
feature_dim=0,
feature_block_grid_size=1,
instance_grid_size=0,
feature_grid_height=None,
feature_grid_width=None,
max_visible_blocks_per_integration=None,
max_support_pixels_per_block_camera=8,
feature_channels_per_thread=8,
max_feature_tile_channels=4096,
feature_integration_kernel='auto',
profile_integration_kernel_timings=False,
accumulator_w_max=1000.0,
device='cuda:0',
max_blocks_override=None,
)

Bases: object

Configuration for volumetric mapping.

Coordinate Convention:

  • extent_meters_xyz: (x, y, z) physical extent of voxel bounds (not centers)

  • grid_shape: (nz, ny, nx) voxel counts - Z is slowest, X is fastest

  • Memory layout: row-major with X contiguous (optimal for image projection)

  • grid_center: world coordinate at center of grid (default: origin)

Note: Actual extent may be slightly larger than requested due to ceil() rounding. Use get_actual_extent() to get the true physical dimensions.

Index-to-World Transform (voxel center):

world_x = grid_center[0] + (ix - (nx-1)/2) * voxel_size world_y = grid_center[1] + (iy - (ny-1)/2) * voxel_size world_z = grid_center[2] + (iz - (nz-1)/2) * voxel_size

Variables:

  • extent_meters_xyz (Tuple[float, float, float]) – Physical extent (x, y, z) of voxel bounds in meters.

  • voxel_size (float) – Voxel edge length in meters.

  • grid_center (torch.Tensor | None) – World coordinate at grid center (default: origin).

  • truncation_distance (float) – TSDF truncation distance in meters.

  • minimum_tsdf_weight (float) – Minimum weight to consider a voxel as observed.

  • depth_minimum_distance (float) – Minimum valid depth in meters.

  • depth_maximum_distance (float) – Maximum valid depth in meters.

  • decay_factor (float) – Weight decay applied to ALL voxels each frame (1.0 = no decay).

  • frustum_decay_factor (float) – Additional decay for in-view voxels.

  • block_size (int) – Voxels per block edge. Supported values are 1 or powers of 2 in [2, 32] (default 8). Specializes the Warp kernel builder; two Mappers with different block sizes can coexist.

  • hash_load_factor (float) – Target hash table load factor.

  • device (str) – CUDA device string.

Parameters:

  • extent_meters_xyz (Tuple[float, float, float])

  • voxel_size (float)

  • esdf_voxel_size (float)

  • extent_esdf_meters_xyz (Tuple[float, float, float])

  • grid_center (torch.Tensor | None)

  • truncation_distance (float)

  • minimum_tsdf_weight (float)

  • depth_minimum_distance (float)

  • depth_maximum_distance (float)

  • decay_factor (float)

  • frustum_decay_factor (float)

  • block_size (int)

  • hash_load_factor (float)

  • roughness (float)

  • color_grid_size (int)

  • seeding_method (str)

  • edt_solver (str)

  • enable_static (bool)

  • static_obstacle_color (Tuple[int, int, int])

  • num_cameras (int)

  • image_height (int | None)

  • image_width (int | None)

  • texture_num_cameras (int | None)

  • texture_camera_image_height (int | None)

  • texture_camera_image_width (int | None)

  • lidar_num_sensors (int)

  • lidar_image_height (int | None)

  • lidar_image_width (int | None)

  • lidar_feature_grid_height (int | None)

  • lidar_feature_grid_width (int | None)

  • lidar_linear_interpolation_max_allowable_difference_vox (float)

  • lidar_nearest_interpolation_max_allowable_dist_to_ray_vox (float)

  • max_visible_blocks_per_lidar_integration (int | None)

  • max_support_pixels_per_block_lidar (int)

  • feature_dim (int)

  • feature_block_grid_size (int)

  • instance_grid_size (int)

  • feature_grid_height (int | None)

  • feature_grid_width (int | None)

  • max_visible_blocks_per_integration (int | None)

  • max_support_pixels_per_block_camera (int)

  • feature_channels_per_thread (int)

  • max_feature_tile_channels (int)

  • feature_integration_kernel (str)

  • profile_integration_kernel_timings (bool)

  • accumulator_w_max (float)

  • device (str)

  • max_blocks_override (int | None)

extent_meters_xyz: Tuple[float, float, float]
voxel_size: float = 0.005
esdf_voxel_size: float = 0.05
extent_esdf_meters_xyz: Tuple[float, float, float] = None
grid_center: torch.Tensor | None = None
truncation_distance: float = 0.04
minimum_tsdf_weight: float = 0.1
depth_minimum_distance: float = 0.1
depth_maximum_distance: float = 10.0
decay_factor: float = 1.0
frustum_decay_factor: float = 1.0
block_size: int = 8

Voxels per block edge. Supported values are 1 or powers of 2 in [2, 32]. See BlockSparseKernels.

hash_load_factor: float = 0.5
roughness: float = 3.0
color_grid_size: int = 1

Number of RGBW control points per block edge.

seeding_method: str = 'gather'

"scatter" or "gather".

  • "scatter": iterates every allocated TSDF voxel and maps each surface voxel to the single ESDF cell that contains its center. The resulting seed band is exactly one ESDF voxel thick at the surface, faithfully matching the TSDF. Launch dimension depends on the number of allocated TSDF blocks, so it is not CUDA-graph safe.

  • "gather": iterates every ESDF voxel and probes 7 TSDF positions (cell center + 6 face centers at ±esdf_vs/2). Because the face-center samples lie on the ESDF cell boundary, they can detect surface voxels that belong to neighboring ESDF cells, producing a dilated seed band (~1.5 voxels thick). This thicker band gives PBA more seed sites near the surface, which typically yields slightly higher collision recall at the cost of not matching the TSDF surface exactly. Fixed launch dimension (D×H×W), CUDA-graph safe.

Type:

ESDF surface seeding strategy

edt_solver: str = 'pba'
enable_static: bool = False
static_obstacle_color: Tuple[int, int, int] = (20, 20, 20)
num_cameras: int = 1
image_height: int | None = None

Camera image height in pixels. Required; used to pre-allocate the camera projective scratch buffer at Mapper construction (buffer size num_cameras * image_height * image_width * num_samples) so no per-frame reallocation or D2H syncs occur.

image_width: int | None = None

Camera image width in pixels. Required; see image_height.

texture_num_cameras: int | None = None

Number of RGB cameras used by projective texture export. Defaults to num_cameras when unset.

texture_camera_image_height: int | None = None

RGB texture camera image height in pixels. Defaults to image_height when unset.

texture_camera_image_width: int | None = None

RGB texture camera image width in pixels. Defaults to image_width when unset.

lidar_num_sensors: int = 0

Number of batched LiDAR range images accepted by the optional LiDAR integrator. 0 disables LiDAR integration.

lidar_image_height: int | None = None

LiDAR range-image height. 1 represents a planar/single-ring scan.

lidar_image_width: int | None = None

LiDAR range-image width / azimuth divisions.

lidar_feature_grid_height: int | None = None

Compile-time LiDAR feature-grid height when both LiDAR and features are enabled.

lidar_feature_grid_width: int | None = None

Compile-time LiDAR feature-grid width when both LiDAR and features are enabled.

lidar_linear_interpolation_max_allowable_difference_vox: float = 2.0

LiDAR bilinear range interpolation discontinuity guard, in voxels.

lidar_nearest_interpolation_max_allowable_dist_to_ray_vox: float = 0.5

LiDAR nearest fallback max distance-to-ray guard, in voxels.

max_visible_blocks_per_lidar_integration: int | None = None

Maximum visible blocks one LiDAR integration frame may process. Defaults to max_blocks when LiDAR is enabled.

max_support_pixels_per_block_lidar: int = 8

Maximum LiDAR support pixels stored per visible block per LiDAR.

feature_dim: int = 0

Per-block feature channel dimensionality. 0 disables features.

feature_block_grid_size: int = 1

Number of feature control points per block edge. This is independent from the incoming 2D feature image dimensions below.

instance_grid_size: int = 0

Number of categorical instance cells per block edge. 0 disables packed instance ownership storage. Deployment uses 4 for 1 cm cells with the default 5 mm voxel and 8-voxel block.

feature_grid_height: int | None = None

Compile-time feature-grid height. Required when feature_dim > 0; must be None when features are disabled.

feature_grid_width: int | None = None

Compile-time feature-grid width. Required when feature_dim > 0; must be None when features are disabled.

max_visible_blocks_per_integration: int | None = None

Maximum visible blocks one integration frame may process. Defaults to max_blocks after that value is computed.

max_support_pixels_per_block_camera: int = 8

Maximum support pixels stored per visible block per camera for RGB and feature integration. Larger values use more scratch memory and preserve more per-block color/feature evidence.

feature_channels_per_thread: int = 8

Number of adjacent feature channels accumulated by one feature-kernel thread. Kept explicit so Python launch grouping and Warp kernel decoding cannot drift apart.

max_feature_tile_channels: int = 4096

Compile-time cap for feature channels accumulated by one tiled feature-kernel CTA. The generated tile width is min(feature_dim, max_feature_tile_channels).

feature_integration_kernel: str = 'auto'

"auto", "grouped", or "tiled". "auto" resolves to the tiled kernel only for feature dimensions and support capacities where benchmarks showed a win.

Type:

Feature integration launch policy

profile_integration_kernel_timings: bool = False

Record per-kernel integration timings into Mapper.get_stats()["last_integration_kernel_timings_ms"]. Disabled by default to avoid profiling synchronizations in normal integration.

accumulator_w_max: float = 1000.0

Upper bound on each RGB or feature-grid node’s accumulator weight. Caps fp16 block_grid_rgb / block_features magnitudes each frame and sets the EMA decay rate for old observations (effective window ~W_max / mean_per_frame_weight). Raise for longer memory, lower for faster adaptation to dynamic scenes.

device: str = 'cuda:0'
max_blocks_override: int | None = None

Explicit block-pool capacity; None keeps the surface-area heuristic.

property grid_shape: Tuple[int, int, int]

Returns (nz, ny, nx) voxel counts.

property max_blocks: int

Return explicit capacity, or use the surface area × thickness heuristic.

TSDF only allocates blocks near surfaces, not throughout the volume. The surface is a volumetric band with thickness = 2 × truncation_distance.

Formula:

max_blocks = surface_area_blocks × thickness_factor × roughness

This scales O(N²) with grid size, not O(N³), matching actual TSDF usage.

property hash_capacity: int

Hash table capacity based on target load factor.

get_actual_extent()

Returns actual physical extent (x, y, z) after ceil() rounding.

Return type:

Tuple[float, float, float]

voxel_to_world(iz, iy, ix)

Convert voxel index to world coordinate at voxel center.

Parameters:

  • iz (int) – Z index (0 to nz-1)

  • iy (int) – Y index (0 to ny-1)

  • ix (int) – X index (0 to nx-1)

Returns:

(world_x, world_y, world_z) coordinate at voxel center.

Return type:

Tuple[float, float, float]

world_to_voxel(
world_x,
world_y,
world_z,
)

Convert world coordinate to voxel index.

Parameters:

  • world_x (float) – World X coordinate.

  • world_y (float) – World Y coordinate.

  • world_z (float) – World Z coordinate.

Returns:

(iz, iy, ix) voxel indices, or (-1, -1, -1) if out of bounds.

Return type:

Tuple[int, int, int]

__init__(
extent_meters_xyz,
voxel_size=0.005,
esdf_voxel_size=0.05,
extent_esdf_meters_xyz=None,
grid_center=None,
truncation_distance=0.04,
minimum_tsdf_weight=0.1,
depth_minimum_distance=0.1,
depth_maximum_distance=10.0,
decay_factor=1.0,
frustum_decay_factor=1.0,
block_size=8,
hash_load_factor=0.5,
roughness=3.0,
color_grid_size=1,
seeding_method='gather',
edt_solver='pba',
enable_static=False,
static_obstacle_color=(20, 20, 20),
num_cameras=1,
image_height=None,
image_width=None,
texture_num_cameras=None,
texture_camera_image_height=None,
texture_camera_image_width=None,
lidar_num_sensors=0,
lidar_image_height=None,
lidar_image_width=None,
lidar_feature_grid_height=None,
lidar_feature_grid_width=None,
lidar_linear_interpolation_max_allowable_difference_vox=2.0,
lidar_nearest_interpolation_max_allowable_dist_to_ray_vox=0.5,
max_visible_blocks_per_lidar_integration=None,
max_support_pixels_per_block_lidar=8,
feature_dim=0,
feature_block_grid_size=1,
instance_grid_size=0,
feature_grid_height=None,
feature_grid_width=None,
max_visible_blocks_per_integration=None,
max_support_pixels_per_block_camera=8,
feature_channels_per_thread=8,
max_feature_tile_channels=4096,
feature_integration_kernel='auto',
profile_integration_kernel_timings=False,
accumulator_w_max=1000.0,
device='cuda:0',
max_blocks_override=None,
)
Parameters:

  • extent_meters_xyz (Tuple[float, float, float])

  • voxel_size (float)

  • esdf_voxel_size (float)

  • extent_esdf_meters_xyz (Tuple[float, float, float])

  • grid_center (torch.Tensor | None)

  • truncation_distance (float)

  • minimum_tsdf_weight (float)

  • depth_minimum_distance (float)

  • depth_maximum_distance (float)

  • decay_factor (float)

  • frustum_decay_factor (float)

  • block_size (int)

  • hash_load_factor (float)

  • roughness (float)

  • color_grid_size (int)

  • seeding_method (str)

  • edt_solver (str)

  • enable_static (bool)

  • static_obstacle_color (Tuple[int, int, int])

  • num_cameras (int)

  • image_height (int | None)

  • image_width (int | None)

  • texture_num_cameras (int | None)

  • texture_camera_image_height (int | None)

  • texture_camera_image_width (int | None)

  • lidar_num_sensors (int)

  • lidar_image_height (int | None)

  • lidar_image_width (int | None)

  • lidar_feature_grid_height (int | None)

  • lidar_feature_grid_width (int | None)

  • lidar_linear_interpolation_max_allowable_difference_vox (float)

  • lidar_nearest_interpolation_max_allowable_dist_to_ray_vox (float)

  • max_visible_blocks_per_lidar_integration (int | None)

  • max_support_pixels_per_block_lidar (int)

  • feature_dim (int)

  • feature_block_grid_size (int)

  • instance_grid_size (int)

  • feature_grid_height (int | None)

  • feature_grid_width (int | None)

  • max_visible_blocks_per_integration (int | None)

  • max_support_pixels_per_block_camera (int)

  • feature_channels_per_thread (int)

  • max_feature_tile_channels (int)

  • feature_integration_kernel (str)

  • profile_integration_kernel_timings (bool)

  • accumulator_w_max (float)

  • device (str)

  • max_blocks_override (int | None)

Return type:

None

get_grid_bounds()

Get world coordinates of grid corners.

Returns:

(min_corner, max_corner) as ((x, y, z), (x, y, z)).

Return type:

Tuple[Tuple[float, float, float], Tuple[float, float, float]]

class MatchedVoxels(
voxels,
block_pool_idx,
block_scores,
)

Bases: object

Result of a feature-similarity query on a BlockSparseTSDF.

Voxels are extracted from the top-K matched blocks. block_pool_idx and block_scores are parallel (K,) tensors in descending score order — index i in one corresponds to index i in the other.

Scores are cosine similarity in the same feature space as the query vector (range [-1, 1]). Callers querying through a projection (e.g., RADIO -> teacher space) must interpret thresholds in the projected space.

block_pool_idx is suitable for direct use with Mapper.clear_blocks to drop the matched blocks from the map.

Variables:

  • voxels (curobo._src.perception.mapper.storage.OccupiedVoxels) – Extracted voxels from the matched blocks. len(voxels) may be 0 even when len(block_pool_idx) > 0 if surface filtering rejected every voxel inside the matched blocks.

  • block_pool_idx (torch.Tensor) – (K,) int32 global pool_idx of matched blocks, sorted by block_scores descending.

  • block_scores (torch.Tensor) – (K,) float32 cosine scores, parallel to block_pool_idx (same descending order).

Parameters:
voxels: curobo._src.perception.mapper.storage.OccupiedVoxels
block_pool_idx: torch.Tensor
block_scores: torch.Tensor
scores_per_voxel(
fill_value=nan,
)

Gather per-voxel score from per-block scores.

Convenience for visualization (e.g., color voxels by similarity). Granularity is fundamentally per-block — every voxel in a matched block shares its block’s score. Prefer block_scores for ranking and filtering.

Parameters:

fill_value (float) – Value written for voxels whose pool_idx is not in block_pool_idx. Defaults to NaN so an unset entry is loud rather than silently masquerading as a low-similarity match. Voxels in voxels should always come from matched blocks; fill_value is a safety net.

Returns:

(N,) float32 tensor parallel to voxels.centers.

Return type:

torch.Tensor

__init__(
voxels,
block_pool_idx,
block_scores,
)
Parameters:
Return type:

None

class OccupiedVoxels(
centers,
block_idx_per_voxel,
block_data,
texture_colors=None,
texture_valid=None,
subvoxel_factor=1,
)

Bases: object

Result of extract_occupied_voxels.

Variables:

  • centers (torch.Tensor) – (N, 3) float32 voxel or subvoxel sample world positions.

  • block_idx_per_voxel (torch.Tensor) – (N,) int32 global pool_idx — index into BlockDataView fields.

  • block_data (curobo._src.perception.mapper.storage.BlockDataView) – Reference view of per-block storage tensors.

  • texture_colors (torch.Tensor | None) – Optional (N, 3) uint8 RGB samples from visibility-tested texture observations.

  • texture_valid (torch.Tensor | None) – Optional (N,) bool mask. True entries use texture_colors when callers prefer texture output.

  • subvoxel_factor (int) – Number of point samples per source voxel axis.

Parameters:
centers: torch.Tensor
block_idx_per_voxel: torch.Tensor
block_data: curobo._src.perception.mapper.storage.BlockDataView
texture_colors: torch.Tensor | None = None
texture_valid: torch.Tensor | None = None
subvoxel_factor: int = 1
instance_state_uint16()

Return packed per-voxel instance owner/confidence as (N,) uint16.

Return type:

torch.Tensor

instance_slots_uint8()

Return categorical instance slots as (N,) uint8.

Return type:

torch.Tensor

instance_confidence_uint8()

Return categorical ownership confidence as (N,) uint8.

Return type:

torch.Tensor

colors_uint8(
eps=1e-06,
prefer_texture=True,
)

Gather per-voxel RGB colors as (N, 3) uint8.

Weighted sums (stored fp16, RGB normalized to [0, 1] at the integration site) are divided in fp32 and rescaled back to [0, 255] before the uint8 cast. Clamp protects degenerate blocks with zero weight from division by zero. When texture colors are present and prefer_texture is true, valid texture samples replace the persistent TSDF color-grid color.

Parameters:
Return type:

torch.Tensor

features(eps=1e-06)

Gather per-voxel normalized features as (N, feature_dim) float32.

Weighted sums (stored fp16) are divided in fp32 to avoid compounding ulp loss. The per-node feature weight is clamped to avoid division by zero for nodes that never received a feature observation.

Parameters:

eps (float)

Return type:

torch.Tensor

__init__(
centers,
block_idx_per_voxel,
block_data,
texture_colors=None,
texture_valid=None,
subvoxel_factor=1,
)
Parameters:
Return type:

None

class OccupiedPoints(
positions,
count,
source_voxels,
_block_data,
)

Bases: object

Borrowed GPU surface samples; only rows below count[0] are valid.

Reused by the next extraction with the same budget. Source indices refer to current mapper storage and are invalid after block recycling/reset. Sampling does not add occupancy evidence, modify TSDF, or provide collision bounds.

Parameters:
positions: torch.Tensor

float32 CUDA [capacity,3], world-frame positions, not voxel centers.

count: torch.Tensor

int32 CUDA [1], number of accepted proposals (no refill).

source_voxels: torch.Tensor

int32 CUDA [capacity,2], (block pool index, local voxel index).

colors_uint8()

Gather CUDA RGB [capacity,3]; only rows below the GPU count are valid.

Return type:

torch.Tensor

features()

Gather normalized CUDA features [capacity,feature_dim] on demand.

Return type:

torch.Tensor

instance_state_uint16()

Gather categorical owner/confidence without interpolating instance IDs.

Return type:

torch.Tensor

__init__(
positions,
count,
source_voxels,
_block_data,
)
Parameters:
Return type:

None

class PoseDetector(geometry, config)

Bases: object

Point-to-plane ICP detector with Huber loss and coarse-to-fine strategy.

Achieves sub-millimeter accuracy (1.3mm translation, 0.03° rotation) on robot tracking. Works with any geometry class that provides sample_surface_points and get_dof methods.

Supported geometry types: - RigidObjectGeometry: Static objects - ArticulatedRobotGeometry: Robots with FK-based point transformation - RobotMesh: New unified mesh class with Warp support

Key features: - Point-to-plane ICP (3× better than point-to-point) - Huber loss for outlier robustness (12× improvement) - Coarse-to-fine with 64 random rotation initializations - Cached surface points and normals for speed

Initialize pose detector.

Parameters:
__init__(geometry, config)

Initialize pose detector.

Parameters:
detect(camera_obs, config)

Detect pose from camera observation.

Parameters:
Returns:

DetectionResult with pose and alignment error.

Return type:

curobo._src.perception.pose_estimation.detection_result.DetectionResult

detect_from_points(
observed_points,
config,
initial_pose=None,
)

Detect pose from pre-segmented 3D points.

This is useful when you already have segmented points (e.g., from a bounding box or semantic segmentation) and don’t need to extract them from a camera observation.

Parameters:

  • observed_points (torch.Tensor) – [N, 3] tensor of observed 3D points.

  • config (torch.Tensor) – Object configuration (joint angles for robot, ignored for rigid).

  • initial_pose (curobo._src.types.pose.Pose | None) – Optional initial pose guess. If provided, only refines from this pose (no random rotation sampling). If None, uses random rotations.

Returns:

Detection result with pose and alignment error.

Return type:

result

class RobotMesh(
vertices,
faces,
device='cuda:0',
kinematics=None,
link_vertex_ranges=None,
link_names=None,
link_vertices_local=None,
)

Bases: object

Unified mesh representation for rigid objects and articulated robots.

Provides mesh data for both ICP-based and SDF-based pose detection: - mesh property: wp.Mesh for SDF queries - sample_surface_points(): returns points + normals for ICP

For articulated robots, call update(joint_angles) to transform vertices and refit BVH. The mesh reference remains constant (CUDA graph safe).

Example

# Rigid object robot_mesh = RobotMesh.from_trimesh(trimesh.load(“object.stl”)) points, normals = robot_mesh.sample_surface_points(1000)

# Articulated robot robot_mesh = RobotMesh.from_kinematics(kinematics) robot_mesh.update(joint_angles) points, normals = robot_mesh.sample_surface_points(1000)

# Use with SDF detector sdf = wp.mesh_query_point(robot_mesh.mesh.id, query_point, max_dist)

Initialize RobotMesh.

Use factory methods from_trimesh() or from_kinematics() instead of calling this constructor directly.

Parameters:

  • vertices (torch.Tensor) – [V, 3] mesh vertices in world frame.

  • faces (torch.Tensor) – [F, 3] face indices.

  • device (str) – CUDA device string.

  • kinematics (Optional[Kinematics]) – Kinematics instance for articulated robots.

  • link_vertex_ranges (Optional[List[Tuple[int, int]]]) – List of (start, end) indices for each link’s vertices.

  • link_names (Optional[List[str]]) – List of link names (for FK lookup).

  • link_vertices_local (Optional[List[torch.Tensor]]) – List of vertices in each link’s local frame.

__init__(
vertices,
faces,
device='cuda:0',
kinematics=None,
link_vertex_ranges=None,
link_names=None,
link_vertices_local=None,
)

Initialize RobotMesh.

Use factory methods from_trimesh() or from_kinematics() instead of calling this constructor directly.

Parameters:
classmethod from_trimesh(
mesh,
device='cuda:0',
)

Create RobotMesh from a trimesh object (rigid object).

Parameters:
Returns:

RobotMesh instance for rigid object.

Return type:

curobo._src.perception.pose_estimation.mesh_robot.RobotMesh

classmethod from_kinematics(
kinematics,
device='cuda:0',
initial_joint_angles=None,
)

Create RobotMesh from a Kinematics instance (articulated robot).

Combines all link meshes into a single mesh. Call update(joint_angles) to transform vertices based on FK.

Parameters:
Returns:

RobotMesh instance for articulated robot.

Return type:

curobo._src.perception.pose_estimation.mesh_robot.RobotMesh

property mesh: warp._src.types.Mesh

Get Warp mesh for SDF queries.

The mesh reference is constant (CUDA graph safe), but vertices are updated in-place when update() is called.

Returns:

wp.Mesh instance.

property mesh_id: warp._src.types.uint64

Get Warp mesh ID for kernel use.

Returns:

Mesh ID as wp.uint64.

property vertices: torch.Tensor

Get current vertices as torch tensor.

Returns:

[V, 3] vertex positions in world frame.

property faces: torch.Tensor

Get face indices.

Returns:

[F, 3] face vertex indices.

property n_vertices: int

Number of vertices in mesh.

property n_faces: int

Number of faces in mesh.

property is_articulated: bool

True if this is an articulated robot mesh.

property current_joint_angles: torch.Tensor | None

Current joint configuration (None for rigid objects).

update(joint_angles)

Update mesh vertices based on joint configuration.

Transforms all link vertices using FK and refits BVH. No-op for rigid objects.

Parameters:

joint_angles (torch.Tensor) – [N_joints] tensor of joint angles.

Return type:

None

sample_surface_points(
n_points,
resample=False,
)

Sample points and normals from mesh surface.

Uses pre-cached sample indices for consistency. Set resample=True to generate new random samples.

Parameters:

  • n_points (int) – Number of points to sample.

  • resample (bool) – If True, regenerate sample indices. If False, reuse cached.

Returns:

[n_points, 3] surface points in world frame. normals: [n_points, 3] surface normals (unit vectors).

Return type:

points

get_dof()

Get degrees of freedom.

Returns:

0 for rigid objects, n_joints for articulated robots.

Return type:

int

get_trimesh()

Get current mesh as trimesh object.

Useful for visualization and debugging.

Returns:

trimesh.Trimesh with current vertex positions.

Return type:

trimesh.base.Trimesh

class RobotSegmenter(
kinematics,
distance_threshold=0.05,
use_cuda_graph=True,
ops_dtype=torch.bfloat16,
)

Bases: object

Segment robot from depth images using collision spheres.

This class computes masks for robot pixels in depth images by checking distances between depth points and robot collision spheres.

Example

from curobo.perception import RobotSegmenter

segmenter = RobotSegmenter.from_robot_file("franka.yml")
mask, filtered_image = segmenter.get_robot_mask(camera_obs, joint_state)

Initialize the robot segmenter.

Parameters:
__init__(
kinematics,
distance_threshold=0.05,
use_cuda_graph=True,
ops_dtype=torch.bfloat16,
)

Initialize the robot segmenter.

Parameters:
static from_robot_file(
robot_file,
collision_sphere_buffer=None,
distance_threshold=0.05,
use_cuda_graph=True,
device_cfg=DeviceCfg(device=device(type='cuda', index=0), dtype=torch.float32, collision_geometry_dtype=torch.float32, collision_gradient_dtype=torch.float32, collision_distance_dtype=torch.float32),
)

Create a RobotSegmenter from a robot configuration file.

Parameters:

  • robot_file (str | Dict) – Path to robot configuration file or dict.

  • collision_sphere_buffer (float | None) – Optional buffer to add to collision spheres.

  • distance_threshold (float) – Distance threshold for segmentation.

  • use_cuda_graph (bool) – Whether to use CUDA graphs.

  • device_cfg (curobo._src.types.device_cfg.DeviceCfg) – Tensor device configuration.

Returns:

Configured RobotSegmenter instance.

Return type:

curobo._src.perception.robot_segmenter.RobotSegmenter

get_pointcloud_from_depth(
camera_obs,
)

Convert depth image to point cloud.

Parameters:

camera_obs (curobo._src.types.camera.CameraObservation) – Camera observation with depth image.

Returns:

Point cloud tensor.

Return type:

torch.Tensor

update_camera_projection(
camera_obs,
)

Update camera projection rays from observation.

Parameters:

camera_obs (curobo._src.types.camera.CameraObservation) – Camera observation with intrinsics.

Return type:

None

get_robot_mask(
camera_obs,
joint_state,
)

Get robot mask from camera observation and joint state.

Assumes 1 robot and batch of depth images, batch of poses.

Parameters:
Returns:

Tuple of (mask, filtered_image).

Raises:

ValueError – If depth image is not 3D (batch, height, width).

Return type:

Tuple[torch.Tensor, torch.Tensor]

get_robot_mask_from_active_js(
camera_obs,
active_joint_state,
)

Get robot mask using active joint state.

Parameters:
Returns:

Tuple of (mask, filtered_image).

Return type:

Tuple[torch.Tensor, torch.Tensor]

property kinematics: curobo._src.robot.kinematics.kinematics.Kinematics

Get the robot kinematics model.

Get the robot base link name.

class RenderedFeatures(
depth,
normals,
valid,
feature_valid,
feature_block_idx,
feature_node_idx,
image_origin,
full_image_shape,
_warp_data,
_feature_valid_storage,
_feature_dim,
_feature_revision,
_feature_revision_getter,
_render_generation,
_render_generation_getter,
_gather_fp32_kernel,
_gather_fp16_kernel,
_single_camera,
_camera_count,
)

Bases: object

Compact feature-grid references produced by a TSDF surface raycast.

Dense feature vectors are gathered lazily with features. The compact references remain valid until the mapper changes or another feature render reuses the renderer’s index buffers.

Variables:

  • depth (torch.Tensor) – (H, W) or (N, H, W) float32 depth image.

  • normals (torch.Tensor) – (H, W, 3) or (N, H, W, 3) float32 normals.

  • valid (torch.Tensor) – Boolean mask for retained TSDF geometry hits.

  • feature_valid (torch.Tensor) – Boolean mask requiring both geometry and positive feature weight.

  • feature_block_idx (torch.Tensor) – Global block-pool index per pixel, or -1 on a miss.

  • feature_node_idx (torch.Tensor) – Local feature-node index per pixel, or -1 on a miss.

  • image_origin (Tuple[int, int]) – Absolute (top, left) origin of the raycast region.

  • full_image_shape (Tuple[int, int]) – Full camera image shape used by the ray model.

Parameters:
depth: torch.Tensor
normals: torch.Tensor
valid: torch.Tensor
feature_valid: torch.Tensor
feature_block_idx: torch.Tensor
feature_node_idx: torch.Tensor
image_origin: Tuple[int, int]
full_image_shape: Tuple[int, int]
features(
image_patch=None,
channels=slice(None, None, None),
dtype=torch.float32,
)

Materialize a spatial patch and contiguous feature-channel slice.

Parameters:

  • image_patch (Tuple[int, int, int, int] | None) – Optional absolute (top, left, height, width) patch contained in the region raycast by Mapper.render_features.

  • channels (slice) – Contiguous feature channel slice with unit step.

  • dtype (torch.dtype) – Output dtype, either torch.float32 or torch.float16.

Returns:

Dense (H, W, C) or (N, H, W, C) feature tensor. Pixels where feature_valid is false contain zero.

Return type:

torch.Tensor

__init__(
depth,
normals,
valid,
feature_valid,
feature_block_idx,
feature_node_idx,
image_origin,
full_image_shape,
_warp_data,
_feature_valid_storage,
_feature_dim,
_feature_revision,
_feature_revision_getter,
_render_generation,
_render_generation_getter,
_gather_fp32_kernel,
_gather_fp16_kernel,
_single_camera,
_camera_count,
)
Parameters:
Return type:

None

class SemanticFilter(
target,
mode='only',
min_similarity=None,
require_argmax=False,
)

Bases: object

Select TSDF geometry using one fused feature target.

The target tensor is treated as immutable. Construct a new filter after changing target values so prepared output masks are refreshed.

Variables:

  • target (torch.Tensor) – Target feature vector with shape (feature_dim,).

  • mode (Literal['only', 'exclude']) – "only" retains matching geometry; "exclude" removes it.

  • min_similarity (float | None) – Optional inclusive cosine-similarity threshold in [-1, 1].

  • require_argmax (bool) – Also require the feature and target argmax channels to match.

Parameters:
target: torch.Tensor
mode: Literal['only', 'exclude'] = 'only'
min_similarity: float | None = None
require_argmax: bool = False
__init__(
target,
mode='only',
min_similarity=None,
require_argmax=False,
)
Parameters:
Return type:

None

class SDFDetectorCfg(
max_iterations=100,
inner_iterations=25,
convergence_threshold=1e-05,
rotation_convergence_threshold=1e-05,
use_cuda_graph=True,
distance_threshold=0.2,
min_valid_ratio=0.1,
use_huber=True,
huber_delta=0.1,
lambda_initial=0.001,
lambda_factor=10.0,
lambda_min=1e-07,
lambda_max=10000000.0,
rho_min=0.25,
n_points=5000,
device_cfg=<factory>,
)

Bases: object

Configuration for SDF-based pose detector (Levenberg-Marquardt optimization).

Uses mesh SDF queries for implicit correspondence and analytic gradients.

Parameters:
max_iterations: int = 100
inner_iterations: int = 25
convergence_threshold: float = 1e-05
rotation_convergence_threshold: float = 1e-05
use_cuda_graph: bool = True
distance_threshold: float = 0.2
min_valid_ratio: float = 0.1
use_huber: bool = True
huber_delta: float = 0.1
lambda_initial: float = 0.001
lambda_factor: float = 10.0
lambda_min: float = 1e-07
lambda_max: float = 10000000.0
rho_min: float = 0.25
n_points: int = 5000
device_cfg: curobo._src.types.device_cfg.DeviceCfg
property max_distance
__init__(
max_iterations=100,
inner_iterations=25,
convergence_threshold=1e-05,
rotation_convergence_threshold=1e-05,
use_cuda_graph=True,
distance_threshold=0.2,
min_valid_ratio=0.1,
use_huber=True,
huber_delta=0.1,
lambda_initial=0.001,
lambda_factor=10.0,
lambda_min=1e-07,
lambda_max=10000000.0,
rho_min=0.25,
n_points=5000,
device_cfg=<factory>,
)
Parameters:
Return type:

None

class SDFPoseDetector(robot_mesh, config=None)

Bases: object

SDF-based pose detector using mesh SDF queries and Levenberg-Marquardt.

Uses ground-truth mesh SDF for implicit correspondence finding. Supports both rigid objects and articulated robots via RobotMesh.

Uses two-pass kernel design for reduced register pressure and better GPU occupancy: - Pass 1: Query mesh SDF (BVH traversal) - Pass 2: Compute Jacobian and block-reduce

Initialize SDF pose detector.

Parameters:

  • robot_mesh (RobotMesh) – RobotMesh instance providing mesh for SDF queries.

  • config (Optional[SDFDetectorCfg]) – Detector configuration.

__init__(
robot_mesh,
config=None,
)

Initialize SDF pose detector.

Parameters:
detect(
camera_obs,
config=None,
initial_pose=None,
)

Detect pose from camera observation.

Parameters:
Returns:

DetectionResult with refined pose.

Return type:

curobo._src.perception.pose_estimation.detection_result.DetectionResult

detect_from_points(
observed_points,
config=None,
initial_pose=None,
)

Detect object pose from pre-segmented 3D points.

Parameters:

  • observed_points (torch.Tensor) – [N, 3] observed points in world frame.

  • config (torch.Tensor | None) – Joint angles for articulated robots (updates mesh). Ignored for rigid.

  • initial_pose (curobo._src.types.pose.Pose | None) – Initial pose estimate. Required for SDF detector (no random sampling).

Returns:

DetectionResult with refined pose.

Return type:

curobo._src.perception.pose_estimation.detection_result.DetectionResult