Zhengyuan Yang
Description: Prompting ChatGPT for Multimodal Reasoning and Action
chatgpt (875) gpt-4 (79) mm-react (2)
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and dense visual signa
MM-ReAct is a system paradigm that composes numerous vision experts with ChatGPT for multimodal reasoning and action.
Flowchart of MM-REACT to enable image understanding with ChatGPT.
Zhengyuan Yang