Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
초록
We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/
저자 (14명)
- Zirui Wu — LinkedIn 검색
- Chen Guo — LinkedIn 검색
- Tianjian Jiang — LinkedIn 검색
- Zeren Jiang — LinkedIn 검색
- Zeming Li — LinkedIn 검색
- Lingchen Yang — LinkedIn 검색
- Longteng Duan — LinkedIn 검색
- Ying Xue — LinkedIn 검색
- Handi Yin — LinkedIn 검색
- Shenyuan Gao — LinkedIn 검색
- Martin R. Oswald — LinkedIn 검색
- Jun Zhang — LinkedIn 검색
- Ping Tan — LinkedIn 검색
- Jie Song — LinkedIn 검색
저자 LinkedIn 변경 추적은 추후 자동화 예정입니다. 현재는 검색 링크를 제공합니다.