PrismWAM: Decoupling State Representations in World Action Models

PrismWAM: Decoupling State Representations
in World Action Models

Tianyu Xin^1*, Wanqi Zhong^1*, Bowen Hu¹, Zhangchi Guo¹, Lijie Wang¹, Yiming Wang¹, Haoran Liao¹, Heyang Yu¹, Xiaoyu Tian¹, Zhiding Yu², Yiming Li^1†

¹Tsinghua University²NVIDIA

^*Equal contribution^†Corresponding author

Film Paper arXiv Code Model BibTeX

#1Overall on RoboTwin C2R
#2Overall on RoboDojo (w/o Agent track)
NoEmbodied action pretraining
+20.0Points Real World OOD vs OpenWAM

TL;DR

Plan the goal in Semantic Space (RGB), roll out the future in Physical Space (Depth), and read Actions off the physical rollout.

Abstract

World Action Models (WAMs) typically represent the world through latent embeddings from pretrained video models, which entangle heterogeneous aspects of the world, from how it looks (e.g., texture and color) to how it is physically configured (e.g., occupancy and pose). This raises an underexplored question: what should constitute the state of a WAM? We argue that the world state should separate how the world appears from how the world evolves under action: the former captures semantic context, while the latter is governed by physical state.

To this end, we propose PrismWAM, which decouples physics from semantics while jointly modeling state and action. PrismWAM factorizes an embodied task into a Semantic Planning → Physical Rollout → Executable Action pipeline: given only the initial RGB observation, semantic planning learns to predict the final goal image, while physical rollout predicts the fine-grained evolution toward this goal in depth space; executable actions are then generated from the depth rollout. Without any embodied action pretraining or additional data augmentation, PrismWAM achieves strong manipulation performance in both simulation and the real world. Notably, PrismWAM even surpasses a broad range of models with large-scale embodied action pretraining (e.g., OpenWAM-α). More broadly, we reframe World Action Models from modeling how the world looks to how it responds to action, paving the way for action-centric world models grounded in occupancy, flow, tactile sensing, proprioception, and beyond.

Watch the 2-minute film

Sound on

Key Insight

Appearance tells you what to act on. Physics tells you what actions change.

World action models jointly predict video and actions, but a video latent mixes how things look with how they are arranged. PrismWAM splits the state like light through a prism: semantic context stays in RGB, physical state is modelled in depth.

Simulator RGB from the head camera, frame 1 of 32 (reach), default appearance: a wooden roller on a white table. — RoboTwin Grab Roller, head camera. Appearance variants are re-rendered from the simulator at the same physical state; their depth is identical.

RoboTwin Grab Roller, head camera. Appearance variants are re-rendered from the simulator at the same physical state; their depth is identical.

How It Works

PrismWAM decouples semantic planning from physical rollout

We instantiate this principle with a Mixture-of-Transformers (MoT) architecture built upon a pretrained video backbone. During training, the semantic planner predicts the target from the initial RGB observation and transfers its task-level knowledge to the physical rollout via attention layers, while the action branch learns to map the predicted physical evolution to continuous actions.

What to do, how the world responds, how to act.

The RGB stream learns the goal from a single intended future frame.

Depth predicts the dense future trajectory; actions are read out of the physics rollout.

Observed RGB guides depth, depth guides action. Actions never attend to RGB directly.

Depth is adapted first, then all streams train jointly. Inference generates no future RGB.

Information flows one way

Each token group reads only what it needs.

Chain attention

PrismWAM architecture. A DiT visual tower takes history and current RGB, a noised sparse future RGB frame (training only), current depth and noised future depth; a DiT action tower, coupled by Mixture-of-Transformers, denoises the future action chunk. Both towers receive the encoded task context. The full figure also shows the chain attention masks for training and inference. — Mixture-of-Transformers: one visual tower for RGB and depth, one action tower.

Results

Results across five benchmarks, without embodied action pretraining

PrismWAM ranks #1 overall on RoboTwin C2R and #2 on RoboDojo (w/o Agent track), ahead of models pretrained on thousands of hours of robot actions.

RoboTwin clean → randomRoboTwin C2R

PrismWAM*

83.67%

#1overall
RoboDojo

PrismWAM*

36.23

#2w/o Agent track

Every benchmark, side by side

PrismWAM
PrismWAM*
Other methods
Extra embodied action pretraining

Five benchmarks at a glance

Custom

RoboDojo score and success rate

* PrismWAM* denotes the exploratory 15B variant, which is initialized from WAN-2.1-14B with additional video-only pretraining.

Robustness

Robust to sensor-like depth noise

Real depth sensors can add measurement noise and miss readings. We corrupt simulated depth at three sensor-like levels, guided by manufacturers' reported error characteristics, and train and evaluate a model at each. At Level 3, the strongest, PrismWAM reaches 68.30% average success on RoboTwin clean → random, 2.45 points below clean depth.

Observed depth

Clean depth from the head camera: a gripper above a bottle lying on the table, coloured from near to far.

Level 1: fine range noise and thin, broken holes along the depth edges.

01500 mm missing

Metric jitter

Range noise that grows with distance

σ at 0 m / 1 m: 36 / 60 mm
Global shift: ±50 mm

Edge missing

Holes along depth edges

Radius: 7 px
Base fraction: 24%

Patch hole

A connected hole for a few frames

Patch size: 1.5%

Success on RoboTwin clean → random

68.30%

Average, Level 3

−2.45 points vs clean depth

Success rate (%) on RoboTwin clean → random by depth setting
Depth	Clean scene	Random scene	Average
Clean depth	87.40	54.10	70.75
Level 1	89.80	52.70	71.25
Level 2	87.60	53.20	70.40
Level 3	86.50	50.10	68.30

Each setting is its own model, trained and evaluated with that depth. This is a separate run from the benchmark in Results (70.57%).

Real Robots

+20.0 points out of distribution

Three tasks on physical arms against OpenWAM, same tasks and trial counts. Out-of-distribution trials change the lighting, add distractors, or alter object appearance, placement or the tablecloth.

Pooled success, all three tasks

From sensor to action

One external camera gives RGB and aligned depth; the depth is refined first. Every PrismWAM branch also reads the language instruction and the measured joint state.

External camera
RGB
Raw depth
LingBot-Depth + causal temporal smoothing
Refined depth
RGB stream Semantic planning
RGB goal t+32 Training only
Depth stream Physical rollout
Action tower Reads depth
Joint commands 30 Hz
Robot arms

External camera One external view supplies RGB and aligned depth. No wrist images are used. RGB Five recent frames up to the current time t. Raw depth Sensor depth in millimetres, aligned to the RGB view. Black pixels have no reading. LingBot-Depth Refines each raw depth frame; causal temporal smoothing then uses past frames only. Refined depth The frame at t is the initial physical state. RGB stream, part of PrismWAM. Semantic planning: reads the RGB history to capture task intent. RGB goal, part of PrismWAM. A training target only. Inference generates no future RGB. Depth stream, part of PrismWAM. Physical rollout. Starts from the observed depth and predicts the depth rollout, guided by the observed RGB. Action tower, part of PrismWAM. Reads the depth rollout and the measured joint state at t. Joint commands Absolute joint commands for the next 32 steps. Robot arms Execute the joint commands. Arrow keys move between stages.

Instruction

Test conditions

Citation

@misc{prism,
  title         = {PrismWAM: Decoupling State Representations in World Action Models},
  author        = {Xin, Tianyu and Zhong, Wanqi and Hu, Bowen and Guo, Zhangchi and Wang, Lijie and Wang, Yiming and Liao, Haoran and Yu, Heyang and Tian, Xiaoyu and Yu, Zhiding and Li, Yiming},
  year          = {2026},
  eprint        = {2610.26564},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.26564}
}

PrismWAM: Decoupling State Representations in World Action Models