Data Visualization

Data Visualization#

Render kinematic-replay MP4s of any GRAIL motion-library directory — works on the retargeting output, the data-export process, and the public-release layout. Single IsaacSim session per batch, so an N-motion render is much faster than spawning a full training process per clip.

The visualization subpackage lives at grail/visualization/. Two CLI wrappers cover the common cases:

Script	Use case
scripts/visualize.sh	Batch — render every motion in a library
scripts/visualize_single.sh	Render one motion from its `robot/<seq>.pkl`

Underneath, both call the same two Python modules: grail.visualization.prepare_vis_shard (motion-lib → trajectory shard) and grail.visualization.batch_render_replay (single IsaacSim session, one MP4 per motion).

Input layout#

The visualizer expects any directory shaped like:

<motion_lib>/
├── robot/<seq>.pkl       required
├── objects/<seq>.pkl     required
├── object_usd/<seq>.usd  required
└── meta/<seq>.pkl        optional (used for table_pos when present)

Three layouts in the wild all satisfy this:

Layout	Producer	Quat convention	Hand DOFs
`data/motion_lib/<name>/`	retargeting pipeline (`grail.retargeting.retarget`)	wxyz	none
`logs_rl/<exp>/exported/step_/shard_/`	data-export pipeline (`grail.data_export.export_successful_rollouts`)	xyzw	`hand_dof_pos: (T, 14)`
`data/<name>/`	released dataset	xyzw	inherited from exported data

Differences between the three are handled by flags on visualize.sh — see Quaternion convention and Hand DOFs.

Output layout#

Everything lands under <motion_lib>/vis/ (kept separate from the release <motion_lib>/video/ which carries the original 2D HOI render, and from the retarget <motion_lib>/videos/ written by older pipelines):

<motion_lib>/vis/
├── <seq>.mp4                    one per motion
├── all_motions_combined.mp4     concat (only when --max_videos > 0)
└── examples_grid.mp4            4×4 or 2×2 grid (only when --max_videos > 0)

Batch render#

conda activate sonic    # or set $GRAIL_SONIC_ENV
export DISPLAY=:1

# Retargeting output — root_rot is wxyz, must override
QUAT_CONVENTION=wxyz bash grail/visualization/scripts/visualize.sh \
    data/motion_lib/pickup_table

# Data-export dir — defaults match (xyzw)
bash grail/visualization/scripts/visualize.sh \
    logs_rl/<exp>/exported/step_010000/shard_0

# Public-release dir — defaults match (xyzw)
bash grail/visualization/scripts/visualize.sh \
    data/pickup_table

Positional arguments (all but the first are optional):

visualize.sh <motion_lib_path> [max_videos] [cam_offset_x,y,z] [quat_convention]

Arg	Default	Notes
`motion_lib_path`	—	Full path (absolute or repo-relative) to the motion library.
`max_videos`	`16`	Cap on number of motions rendered. Pass `0` to render all motions; `0` also skips the post-processing concat / grid.
`cam_offset`	`1.5,-1.5,1.0`	Camera position relative to the motion centroid. Comma-separated, no spaces.
`quat_convention`	`xyzw`	One of `auto`, `wxyz`, `xyzw`. See Quaternion convention.

Env-var fallbacks: QUAT_CONVENTION (default xyzw).

Single-motion render#

bash grail/visualization/scripts/visualize_single.sh \
    data/pickup_table/robot/pickup_table__apple_0__000.pkl

Takes the full path to a robot/<seq>.pkl and writes <motion_lib>/vis/<seq>.mp4. Useful for spot-checking a single clip without re-rendering the whole library.

visualize_single.sh <robot_pkl_path> [cam_offset_x,y,z] [quat_convention]

Quaternion convention#

root_rot is stored in two different conventions across the codebase:

grail/retargeting/retarget.py writes wxyz.
grail/data_export/export_successful_rollouts.py writes xyzw.

The renderer expects wxyz, so prepare_vis_shard.py canonicalizes before passing data downstream. Pick the right setting via --quat_convention (CLI) or QUAT_CONVENTION (env var):

Value	Behavior	Use for
`xyzw` (default)	Always swap `[x,y,z,w] → [w,x,y,z]` before rendering	data-export dir + public-release dirs
`wxyz`	No-op pass-through	retargeting output (`data/motion_lib/<name>/`)
`auto`	Magnitude-based detection: whichever of slot 0 / slot 3 carries more energy is taken to be the scalar `w`	Mixed corpora; not robust to motions that don’t start near-upright (e.g. backward-leaning)

prepare_vis_shard.py prints a per-batch summary line:

quat conventions (root_rot): wxyz=0 xyzw=16

so you can confirm the choice from the log.

Hand DOFs#

The gripper renders from hand_dof_pos: (T, 14) in the input robot/<seq>.pkl when present, otherwise the renderer zero-pads the missing 14 DOFs, so the gripper stays in its open pose.

Status appears in the Step-1 log:

hand DOFs from hand_dof_pos: 16 / 16

Under the hood#

visualize.sh runs in three logical steps:

1. prepare_vis_shard.py   motion_lib/{robot,objects,meta} → /tmp/vis_shard_<key>/
                          (per-motion trajectory.pkl + synthetic metrics_eval.json)
2. batch_render_replay.py one IsaacSim session, hot-swap object USDs between motions
                          → <motion_lib>/vis/<seq>.mp4
3. (optional) ffmpeg      add per-clip labels, concat into all_motions_combined.mp4,
                          build 4×4 (or 2×2 if fewer than 16) examples_grid.mp4

Step 2 also handles MuJoCo → IsaacLab DOF reordering for the 29 G1 body joints and pads to 43 DOFs (zero-fill) when hand_dof_pos is absent.

You can call the Python modules directly if you want to script around the shell wrapper:

python -m grail.visualization.prepare_vis_shard \
    --data_dir <motion_lib> --shard_dir /tmp/shard --max_motions 16 \
    --quat_convention xyzw

python -u -m grail.visualization.batch_render_replay \
    --shard_dir /tmp/shard --traj_dir /tmp/shard/trajectories \
    --object_usd_dir <motion_lib>/object_usd \
    --output_dir <motion_lib>/vis \
    --camera_offset 1.5 -1.5 1.0 --camera_target 0.0 0.0 0.8 \
    --skip_existing --headless

Following up with the web visualizer#

<motion_lib>/vis/*.mp4 is exactly the input the web visualizer consumes for hover-to-play previews — generate vis/ first, then point grail.web_visualizer.generate_manifest at the same motion library.

Troubleshooting#

Symptom	Likely cause / fix
Robot is rotated horizontally / lying on its side	Wrong `quat_convention`. Retarget pkls are `wxyz`; data-export and release pkls are `xyzw`. Match it explicitly instead of relying on `auto`.
Gripper stays open even at the grasp moment	`robot/<seq>.pkl` lacks `hand_dof_pos`. Re-run the data-export step with the post-2026-06-01 code (it persists the full 14-DOF hand trajectory alongside the scalar averages).
`Error: required subdir missing: <path>/object_usd`	The directory isn’t a motion-library layout. Verify `robot/`, `objects/`, `object_usd/` all exist; `meta/` is optional.
`[DOF] Padding 29-DOF trajectories to 43-DOF articulation` printed and the gripper looks open	Expected when `hand_dof_pos` is absent — the renderer pads with zeros (= open pose). Not an error.
Render hangs on first motion	IsaacSim init failed silently. Confirm `DISPLAY` is set and the `sonic` env activated. The renderer has a 180 s watchdog that force-exits if no per-frame heartbeat fires, so stalls do eventually self-terminate.
Black mujoco viewer / `glfwInit failed`	`export DISPLAY=:1` before activating conda (same as retargeting — `DISPLAY` is an env var, not a conda setting).