OpenEQA: Embodied Question Answering in the Era of Foundation Models
Citations Over TimeTop 10% of 2024 papers
Abstract
We present a modern formulation of Embodied Question Answering (EQA) as the task of understanding an environment well enough to answer questions about it in natural language. An agent can achieve such an understanding by either drawing upon episodic memory, exemplified by agents on smart glasses, or by actively exploring the environment, as in the case of mobile robots. We accompany our formulation with OpenEQA - the first open-vocabulary benchmark dataset for EQA supporting both episodic memory and active exploration use cases. OpenEQA contains over 1600 high-quality human generated questions drawn from over 180 real-world environments. In addition to the dataset, we also provide an automatic LLM-powered evaluation protocol that has excellent correlation with human judgement. Using this dataset and evaluation protocol, we evaluate several state-of-the-art foundation models including GPT-4V, and find that they significantly lag behind human-level performance. Consequently, OpenEQA stands out as a straightforward, measurable, and practically rele-vant benchmark that poses a considerable challenge to current generation offoundation models. We hope this inspires and stimulates future research at the intersection of Embod-ied AI, conversational agents, and world models.
Related Papers
- → Simulating sensorimotor metaphors: Novel metaphors influence sensory judgments(2014)98 cited
- The View of Embodied Emotion: A New Perspective on Emotion Study(2011)
- An Embodied-philosophical Account of Computer-assisted Language Education(2010)
- → THE EMBODIED COGNITION IN KINDERGARTEN: A PRE-WRITING EXPERIENCE WITH AN EMBODIED AND ANALOGICAL APPROACH(2019)
- → Critically Considering Embodied Cognition and Research in Theatre and Performance(2022)