Embodied AI Workshop
Unnat Jain
Changan Chen
Sagnik Majumder
Chen Gao
Description: Moving towards embodied agents that can move around and understand the space with visual and auditory perception ---
fair (1472) audio visual (545) uiuc (45) embodied (42) ut austin (23) soundspaces (1) sound spaces (1) av navigation (1)
Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf -- restricted to solely their visual perception of the environment. This project aims to fill this void by building agents capable of audio-visual navigation in complex, acoustically and visually realistic 3D environments.
SoundSpaces is a first-of-its-kind dataset of audio renderings based on geometrical acoustic simulations for two sets of publicly available 3D environments -- Matterport3D 1 and Replica 2 . SoundSpaces is AIHabitat-compatible and allows rendering arbitrary sounds at any pair of source and receiver (agent) locations on a uniform grid of nodes. The room-impulse-responses (RIR) enable realistic audio experience of arbitrary sounds in the photorealistic environments. Click on the gif to view the demonstration v
UT Austin is supported in part by DARPA Lifelong Learning Machines. We thank Alexander Schwing, Dhruv Batra, Erik Wijmans, Oleksandr Maksymets, Ruohan Gao, and Svetlana Lazebnik for valuable discussions and support with the AI-Habitat platform. We also thank Abhishek Das for sharing the website code for visualdialog.org.
Embodied AI Workshop
Unnat Jain
Changan Chen
Sagnik Majumder
Chen Gao