Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology
초록
We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations. We introduce RefRad2D, a large-scale bilingual (German/English) dataset of 1.2M CT and MR image-text pairs derived from clinical practice, with task-specific VQA and spatial grounding subsets generated automatically via LLM-based curation and automated segmentation. Trained on this data, our model RadGrounder jointly performs report generation, visual question answering, and spatial grounding via bounding-box detection or segmentation. On external VQA benchmarks (Slake, VQA-RAD), RadGrounder achieves competitive results with specialized medical VLMs. Adding our clinical data to the training mixture improves open-ended VQA over fine-tuning on the downstream datasets alone, showing the transferability of our dataset. Crucially, adding grounding supervision does not degrade language quality, enabling spatially verifiable outputs at no cost to VQA performance.
저자 (7명)
- Yusuf Salcan — LinkedIn 검색
- Simon Ging — LinkedIn 검색
- Robin Schirrmeister — LinkedIn 검색
- Philipp Arnold — LinkedIn 검색
- Elmar Kotter — LinkedIn 검색
- Behzad Bozorgtabar — LinkedIn 검색
- Thomas Brox — LinkedIn 검색
저자 LinkedIn 변경 추적은 추후 자동화 예정입니다. 현재는 검색 링크를 제공합니다.