Andy Fang
Gabriel Sarch
Description: TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors
We introduce TIDEE, an embodied agent that tidies up a disordered scene based on learned commonsense object placement and room arrangement priors. TIDEE explores a home environment, detects objects that are out of their natural place, infers plausible object contexts for them, localizes such contexts in the current scene, and repositions the objects. Commonsense priors are encoded in three modules: i) visuo-semantic detectors that detect out-of-place objects, ii) an associative neural graph memory of object
Human evaluations on the resulting room reorganizations show TIDEE outperforms ablative versions of the model that do not use one or more of the commonsense priors. On a related room rearrangement benchmark that allows the agent to view the goal state prior to rearrangement, a simplified version of our model significantly outperforms a top-performing method by a large margin.
TIDEE can clean up never-before-seen rooms without any instruction or previous exposure of the room and object instances. TIDEE does this by exploring the scene, detecting objects and classifying whether they are in place or out of place. If an object is out of place, TIDEE uses graph inference in its joint external graph memory and scene graph to infer plausible receptacle categories. It then explores the scene guided by a visual search network that suggests where a receptacle category may be found, given
Andy Fang
Gabriel Sarch