MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents
초록
Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.
저자 (12명)
- Huanyao Zhang — LinkedIn 검색
- Jiepeng Zhou — LinkedIn 검색
- Jiaye Lin — LinkedIn 검색
- Ontai Tang — LinkedIn 검색
- Jingtian Hao — LinkedIn 검색
- Mugeng Liu — LinkedIn 검색
- Ru Peng — LinkedIn 검색
- Lang Mei — LinkedIn 검색
- Xiaohan Yu — LinkedIn 검색
- Liyan Liu — LinkedIn 검색
- Chong Chen — LinkedIn 검색
- Wentao Zhang — LinkedIn 검색
저자 LinkedIn 변경 추적은 추후 자동화 예정입니다. 현재는 검색 링크를 제공합니다.