Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering
초록
Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D
저자 (10명)
- Hsiang-Wei Huang — LinkedIn 검색
- Fu-Chen Chen — LinkedIn 검색
- Li-Wu Tsao — LinkedIn 검색
- Cheng-Han Lee — LinkedIn 검색
- Che-Chun Su — LinkedIn 검색
- Lu Xia — LinkedIn 검색
- Ronghui Peng — LinkedIn 검색
- Jenq-Neng Hwang — LinkedIn 검색
- Min Sun — LinkedIn 검색
- Cheng-Hao Kuo — LinkedIn 검색
저자 LinkedIn 변경 추적은 추후 자동화 예정입니다. 현재는 검색 링크를 제공합니다.