Baoxiong Jia @ UCLA
Yixin Chen
Description: SceneVerse: Scaling 3D Vision-Language Learnding for Grounded Scene Understanding
3d vision-language learning (1) scaling effect (1) grounded scene understanding (1)
✶ indicates equal contribution
TL;DR We propose SceneVerse, the first million-scale 3D vision-language dataset with 68K 3D indoor scenes and 2.5M vision-language pairs. We demonstrate the scaling effect by (i) achieving state-of-the-art on all existing 3D visual grounding benchmarks and (ii) showcasing zero-shot transfer capabilities with our GPS (Grounded Pre-training for Scenes) model.
3D vision-language grounding, which focuses on aligning language with the 3D physical environment, stands as a cornerstone in the development of embodied agents. In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii
Baoxiong Jia @ UCLA
Yixin Chen