T2-z: Test-Time Optimization of Environment Latents for Robotic World Models
초록
Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-Z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take. During training, T2-Z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations. Across 7 simulated settings and 4 real-world robotic tasks, T2-Z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-Z can efficiently infer environment latents from as little as a single test-time interaction trajectory.
저자 (12명)
- Chenyuan Zhou — LinkedIn 검색
- Yining Hong — LinkedIn 검색
- Huang Huang — LinkedIn 검색
- Yunfan Jiang — LinkedIn 검색
- Wenlong Huang — LinkedIn 검색
- David Held — LinkedIn 검색
- Vitor Campagnolo Guizilini — LinkedIn 검색
- Yue Wang — LinkedIn 검색
- Li Fei-Fei — LinkedIn 검색
- Jiajun Wu — LinkedIn 검색
- Leonidas Guibas — LinkedIn 검색
- Yejin Choi — LinkedIn 검색
저자 LinkedIn 변경 추적은 추후 자동화 예정입니다. 현재는 검색 링크를 제공합니다.