Prediction overhead
Besides the actions, each action chunk requires denoising the far more numerous video tokens over several steps. A synchronous robot stands still during each request.
World action model · real-time robot control
Efficient World Action Model with Hybrid Memory
A 4.5B world action model with minute-level memory that runs in real time on edge devices.
1NVIDIA 2UCSD 3THU 4MIT 5HKU 6Inco AI
*Equal contribution †Core contributor
01 · Why
World action models (WAMs) leverage the world knowledge of video generation models to improve action generation. Two problems keep them from serving as the low-level policy that must run stably on edge devices.
Besides the actions, each action chunk requires denoising the far more numerous video tokens over several steps. A synchronous robot stands still during each request.
Many manipulation tasks depend on information that is no longer visible, such as the number of completed steps or the position of an occluded object. Existing WAMs either act from the current observation alone or keep only a few sparsely sampled past frames, which miss most of the episode and add to the latency as the memory grows.
02 · Method
SANA-Robot is a hybrid linear diffusion transformer trained from Sana-Video 2. It stacks 8 blocks, each with three Gated DeltaNet (GDN) layers followed by one softmax attention layer. A GDN layer compresses the past into a fixed-size recurrent state, so its cost does not grow with the context. A softmax layer attends to every token exactly, at a cost that grows with the context.
Attention is causal across action chunks, so each chunk reads the past but never the future. The softmax layers attend to the first frame and to a sliding window over the last few chunks, through which information propagates chunk by chunk. The GDN layers absorb every past chunk into their fixed-size state and keep a memory of the whole episode. Each chunk is written into the memory once, so every action chunk has the same cost.
What the current chunk reads from the past
Success on the memory tasks of RoboDojo (%), our own evaluation
The baseline memories are used to post-train SANA-Robot-base with the same data and initialization. All three are evaluated in our own runs rather than in the official evaluation, so the number of the hybrid memory differs from its official result in the benchmark table under Results.Play the episode or click the timeline. The ball is visible only in the first two seconds; when the robot reaches for a cup, the answer lies about ten seconds in the past. Chunks are drawn every 32 actions at 30 Hz (about 1.07 s). The clip illustrates the mechanism and is not aligned to the actual requests of the policy.
Each chunk stores the video latent of its full span of 32 control steps, so the memory keeps every position the target cup passes through in the Shell Game, which sparsely sampled frames may miss.
The targets read the current observation, but the observation does not read them and is not written into the memory. With bidirectional attention the policy takes a shortcut through the current frame and ignores the memory (9.1% on the memory tasks, against 48.8%).
The actions barely read the history directly. The video predicted for the current chunk reads the history, and the actions read this video, which is also why the memory stores only video latents.
All three modes share one token sequence of multi-view video, proprioception and actions, and differ only in which tokens are clean conditions and which are noised targets. One network thus learns all three roles from the same data: 19.4K hours from 23 datasets of robots, handheld grippers, simulation and human video.
The memory changes both training and deployment, and the infrastructure of SANA-Robot is designed around it.
Training
The memory must be learned on whole episodes, whose lengths vary widely. Multimodal sequence packing with task-balanced ordering trains on whole episodes without padding and with a stable task mix in every step. Balanced and shuffled data loading further mixes many datasets in every step.
Deployment
Memory RTC extends real-time chunking to a policy with memory by keeping one memory per execution phase, so that every request reads a memory laid out as in training. Fused kernels, CUDA graphs, prefix caching and low-bit GEMMs reduce the latency by 5.3× to 115 ms per action chunk on an RTX 5090. See it on the robot ↓
03 · Results
Five real-world tasks, each with 100 demonstrations and 20 evaluation rollouts. The baselines, π0.5 and OpenWAM-α, are fine-tuned on the same demonstrations of each task.
Long horizon · precision
Seven rings are stacked one after another, and each must be lowered with its hole aligned to the peg. An error at one step compromises all the steps after it.
Rings stacked consecutively in the correct order before the first error, out of 7 (%).Long horizon
Seven sub-tasks on a Galaxea R1, from picking up the capsule to handing over the coffee cup.
Finished steps before the first error (%).Precision
Each turn must grasp one layer of the cube and rotate it without moving the others.
Successful turns of the cube (%).Long memory
A ball is hidden under one of three cups and the cups are shuffled, after which the policy must lift the cup that hides the ball. SANA-Robot-AR, with a window of N = 3, succeeds in 75% of the rollouts. SANA-Robot-base sees only the current observation and succeeds in 30%, no better than chance.
Rollouts in which the robot picks the cup that hides the ball (%).Fast reaction
The robot must react to a fast-moving puck, so every pause between chunks costs hits. Asynchronous execution removes the stall before each chunk.
Attempts in which the robot hits the puck (%). Asynchronous with FP8 GEMMs, as in the clip; synchronous with BF16. See Real time.Latency per action chunk against the score on the memory dimension of RoboDojo, all measured on one RTX 5090.
04 · Real time
Real-time closed-loop control requires the robot to run stably at its control frequency, 30 Hz in our deployment, without any idle time. The pipeline must be asynchronous, since in synchronous execution the robot idles during every request, and inference must be fast, so that each chunk is planned from a more recent observation.
BF16 · Synchronous
45%of the pucks hit
The robot stalls for 181.3 ms before each chunk while the model runs.
FP8 · Asynchronous
60%of the pucks hit
Inference overlaps the execution of the current chunk. The stall falls to about 4 ms, the network latency.
NVFP4 · Asynchronous
50%of the pucks hit
Two more missed hits in 20 attempts than BF16 and FP8, with a 30% lower latency than BF16.
Asynchronous execution with BF16 GEMMs and d = 8: the request is issued when 8 actions of the current chunk remain and runs while the robot executes them.
FP8 and NVFP4 GEMMs quantize the weights and activations of the backbone to 8 and 4 bits, and cut the latency on RTX PRO 6000 by 20% and 30% over BF16.
Tower of Hanoi (%)
Air Hockey (%)
In training, a chunk that starts at step t reads memory chunks of C = 32 steps that end at t, t − C, t − 2C and so on. In asynchronous execution, consecutive requests are only C − d steps apart. A single memory can then serve all requests only by breaking this chunk structure. Memory RTC instead keeps one memory per phase φ = t mod C, and each request reads the memory of its own phase.
The animation runs along the control steps: every window of observed video is written into a memory once, and each request reads while the robot keeps moving. Drag across the chart to scrub, or change d. The error is the open-loop end-effector position error of asynchronous execution relative to synchronous execution (C = 32), on 70 RoboDojo episodes. Memory RTC trades one copy of the memory per phase for exact training-time memories, without recomputing the history at any request.
Per-request latency of SANA-Robot-AR (N = 1) in synchronous execution with K = 4 denoising steps and three camera views: the sum of video encoding, prefill (including the memory) and denoising. Optimized: kernel fusion, CUDA graphs and prefix caching in BF16. Under the real-time condition, a 30 Hz control loop is met with d = 8 on RTX PRO 6000 and RTX 5090, and with d = 16 on Jetson Thor.
05 · World simulator
In the FDM mode, SANA-Robot predicts the future video caused by given actions. A policy reads the generated observation and proposes the next action chunk, the simulator renders its visual consequences, and the loop repeats entirely in imagination.
Success rate of three policies on seven tasks in the world model against the real robot. A faithful simulator puts the points on y = x.
Citation
@article{zhao2026sanarobot,
title = {SANA-Robot: Efficient World Action Model with Hybrid Memory},
author = {Zhao, Yuyang and Li, Zekai and Pan, Yicheng and Tang, Jiaming and Zhang, Zhuoyang and Xing, Jiaxu and Reuss, Moritz and Peng, Quanquan and Chen, Junsong and Xie, Enze and Liu, Zhijian and Han, Song},
year = {2026}
}