World action model · real-time robot control

SANA-Robot

Efficient World Action Model with Hybrid Memory

A 4.5B world action model with minute-level memory that runs in real time on edge devices.

Yuyang Zhao1*†, Zekai Li1,2*†, Yicheng Pan3†, Jiaming Tang4, Zhuoyang Zhang4, Jiaxu Xing1, Moritz Reuss1, Quanquan Peng2, Junsong Chen1,5, Enze Xie1, Zhijian Liu2,6, Song Han1,4

1NVIDIA   2UCSD   3THU   4MIT   5HKU   6Inco AI
*Equal contribution   †Core contributor

SANA-Robot in 1:47. Five real-world tasks, the hybrid memory, memory RTC and the results. The video starts muted; turn the sound on for the music. Music: 'Growing Up' by Scott Buckley (edited) - released under CC-BY 4.0. www.scottbuckley.com.au
4.5Bparameters in one network that is a policy, an inverse and a forward dynamics model
≥ 1 minof memory over the whole episode, at the same cost for every action chunk
115 msper action chunk on an RTX 5090 with NVFP4 GEMMs, 135 ms with FP8
30 Hzclosed-loop, asynchronous robot control on edge devices

01 · Why

Video prediction makes world action models slow, and most of them cannot remember.

World action models (WAMs) leverage the world knowledge of video generation models to improve action generation. Two problems keep them from serving as the low-level policy that must run stably on edge devices.

Prediction overhead

0.6 sper request for our 4.5B model on an RTX 5090, without optimization
< 0.1 sfor π0.5, a vision-language-action model

Besides the actions, each action chunk requires denoising the far more numerous video tokens over several steps. A synchronous robot stands still during each request.

No long-term memory

30%on the Shell Game for SANA-Robot-base, which sees only the current observation: no better than chance
75%for SANA-Robot-AR, with the hybrid memory

Many manipulation tasks depend on information that is no longer visible, such as the number of completed steps or the position of an occluded object. Existing WAMs either act from the current observation alone or keep only a few sparsely sampled past frames, which miss most of the episode and add to the latency as the memory grows.

02 · Method

One hybrid transformer, three core designs.

SANA-Robot is a hybrid linear diffusion transformer trained from Sana-Video 2. It stacks 8 blocks, each with three Gated DeltaNet (GDN) layers followed by one softmax attention layer. A GDN layer compresses the past into a fixed-size recurrent state, so its cost does not grow with the context. A softmax layer attends to every token exactly, at a cost that grows with the context.

The video VAE encodes each camera view separately into the observation latent o and the future video latent z, which enter the DiT in one sequence with the proprioception s and the actions a. The Policy mode is shown: z and a are noised targets and are denoised jointly. SANA-Robot-AR adds the hybrid memory.
1

Hybrid memory at constant cost

Attention is causal across action chunks, so each chunk reads the past but never the future. The softmax layers attend to the first frame and to a sliding window over the last few chunks, through which information propagates chunk by chunk. The GDN layers absorb every past chunk into their fixed-size state and keep a memory of the whole episode. Each chunk is written into the memory once, so every action chunk has the same cost.

Three-Cup Shell Game · real robot · 1×

What the current chunk reads from the past

episode starttime →
Softmax window N
The Shell Game policy uses N = 3. From N = 1 to N = 4 the latency rises by only 2.5 ms (173.3 → 175.8 ms, BF16, RTX 5090).

Success on the memory tasks of RoboDojo (%), our own evaluation

The baseline memories are used to post-train SANA-Robot-base with the same data and initialization. All three are evaluated in our own runs rather than in the official evaluation, so the number of the hybrid memory differs from its official result in the benchmark table under Results.

Play the episode or click the timeline. The ball is visible only in the first two seconds; when the robot reaches for a cup, the answer lies about ten seconds in the past. Chunks are drawn every 32 actions at 30 Hz (about 1.07 s). The clip illustrates the mechanism and is not aligned to the actual requests of the policy.

Fine-grained, chunk-level

Each chunk stores the video latent of its full span of 32 control steps, so the memory keeps every position the target cup passes through in the Shell Game, which sparsely sampled frames may miss.

Read, but never written

The targets read the current observation, but the observation does not read them and is not written into the memory. With bidirectional attention the policy takes a shortcut through the current frame and ignores the memory (9.1% on the memory tasks, against 48.8%).

The predicted video is the bridge

The actions barely read the history directly. The video predicted for the current chunk reads the history, and the actions read this video, which is also why the memory stores only video latents.

2

A unified world action model

All three modes share one token sequence of multi-view video, proprioception and actions, and differ only in which tokens are clean conditions and which are noised targets. One network thus learns all three roles from the same data: 19.4K hours from 23 datasets of robots, handheld grippers, simulation and human video.

clean token: condition noised token: target that carries the loss
3

Full-stack infrastructure for memory

The memory changes both training and deployment, and the infrastructure of SANA-Robot is designed around it.

Training

Whole episodes, without padding

The memory must be learned on whole episodes, whose lengths vary widely. Multimodal sequence packing with task-balanced ordering trains on whole episodes without padding and with a stable task mix in every step. Balanced and shuffled data loading further mixes many datasets in every step.

Deployment

Memory RTC and a 5.3× faster request

Memory RTC extends real-time chunking to a policy with memory by keeping one memory per execution phase, so that every request reads a memory laid out as in training. Fused kernels, CUDA graphs, prefix caching and low-bit GEMMs reduce the latency by 5.3× to 115 ms per action chunk on an RTX 5090. See it on the robot ↓

03 · Results

Long-horizon, precise, and it remembers.

Five real-world tasks, each with 100 demonstrations and 20 evaluation rollouts. The baselines, π0.5 and OpenWAM-α, are fine-tuned on the same demonstrations of each task.

real robot · 1×

Long horizon · precision

Tower of Hanoi

Seven rings are stacked one after another, and each must be lowered with its hole aligned to the peg. An error at one step compromises all the steps after it.

Rings stacked consecutively in the correct order before the first error, out of 7 (%).
real robot · 1×

Long horizon

Make Coffee

Seven sub-tasks on a Galaxea R1, from picking up the capsule to handing over the coffee cup.

Finished steps before the first error (%).
real robot · 1×

Precision

Magic Cube

Each turn must grasp one layer of the cube and rotate it without moving the others.

Successful turns of the cube (%).
real robot · 1×
real robot · 1×

Long memory

Three-Cup Shell Game

A ball is hidden under one of three cups and the cups are shuffled, after which the policy must lift the cup that hides the ball. SANA-Robot-AR, with a window of N = 3, succeeds in 75% of the rollouts. SANA-Robot-base sees only the current observation and succeeds in 30%, no better than chance.

Rollouts in which the robot picks the cup that hides the ball (%).
real robot · 1×

Fast reaction

Air Hockey

The robot must react to a fast-moving puck, so every pause between chunks costs hits. Asynchronous execution removes the stall before each chunk.

Attempts in which the robot hits the puck (%). Asynchronous with FP8 GEMMs, as in the clip; synchronous with BF16. See Real time.

Simulation benchmarks

Speed against memory

Latency per action chunk against the score on the memory dimension of RoboDojo, all measured on one RTX 5090.

04 · Real time

115 ms per action chunk, and 30 Hz closed-loop control.

Real-time closed-loop control requires the robot to run stably at its control frequency, 30 Hz in our deployment, without any idle time. The pipeline must be asynchronous, since in synchronous execution the robot idles during every request, and inference must be fast, so that each chunk is planned from a more recent observation.

Air Hockey: every pause between chunks costs hits

real robot · 1×

BF16 · Synchronous

45%of the pucks hit

The robot stalls for 181.3 ms before each chunk while the model runs.

real robot · 1×

FP8 · Asynchronous

60%of the pucks hit

Inference overlaps the execution of the current chunk. The stall falls to about 4 ms, the network latency.

real robot · 1×

NVFP4 · Asynchronous

50%of the pucks hit

Two more missed hits in 20 attempts than BF16 and FP8, with a 30% lower latency than BF16.

Why asynchronous execution keeps 30 Hz

Asynchronous execution with BF16 GEMMs and d = 8: the request is issued when 8 actions of the current chunk remain and runs while the robot executes them.

Low-bit inference keeps the success

FP8 and NVFP4 GEMMs quantize the weights and activations of the backbone to 8 and 4 bits, and cut the latency on RTX PRO 6000 by 20% and 30% over BF16.

Tower of Hanoi (%)

Air Hockey (%)

Asynchronous execution with memory: direct RTC against memory RTC

In training, a chunk that starts at step t reads memory chunks of C = 32 steps that end at t, t − C, t − 2C and so on. In asynchronous execution, consecutive requests are only C − d steps apart. A single memory can then serve all requests only by breaking this chunk structure. Memory RTC instead keeps one memory per phase φ = t mod C, and each request reads the memory of its own phase.

Asynchronous delay d (actions dropped per chunk)

The animation runs along the control steps: every window of observed video is written into a memory once, and each request reads while the robot keeps moving. Drag across the chart to scrub, or change d. The error is the open-loop end-effector position error of asynchronous execution relative to synchronous execution (C = 32), on 70 RoboDojo episodes. Memory RTC trades one copy of the memory per phase for exact training-time memories, without recomputing the history at any request.

Latency per request, from server to edge

Per-request latency of SANA-Robot-AR (N = 1) in synchronous execution with K = 4 denoising steps and three camera views: the sum of video encoding, prefill (including the memory) and denoising. Optimized: kernel fusion, CUDA graphs and prefix caching in BF16. Under the real-time condition, a 30 Hz control loop is met with d = 8 on RTX PRO 6000 and RTX 5090, and with d = 16 on Jetson Thor.

05 · World simulator

The same network is also a world simulator.

In the FDM mode, SANA-Robot predicts the future video caused by given actions. A policy reads the generated observation and proposes the next action chunk, the simulator renders its visual consequences, and the loop repeats entirely in imagination.

Policy evaluation on DROID

Success rate of three policies on seven tasks in the world model against the real robot. A faithful simulator puts the points on y = x.

66.8EWMScore-P on WorldArena, second on the video-quality leaderboard and the highest visual quality
0.3 sto generate 30 frames at 640 × 480 on one H100 with the 4-step student, which keeps a score of 66.2
58.3%mean simulated success over the 21 policy and task pairs, against 52.6% on the real robot and 32.4% in Ctrl-World

Citation

BibTeX

@article{zhao2026sanarobot,
  title   = {SANA-Robot: Efficient World Action Model with Hybrid Memory},
  author  = {Zhao, Yuyang and Li, Zekai and Pan, Yicheng and Tang, Jiaming and Zhang, Zhuoyang and Xing, Jiaxu and Reuss, Moritz and Peng, Quanquan and Chen, Junsong and Xie, Enze and Liu, Zhijian and Han, Song},
  year    = {2026}
}