ROME: Evaluating pre-trained vision-language models on reasoning beyond visual common sense

• Main Authors: ZHOU, Kankan, LAI, Eason, YEONG, Au Wei Bin, MOURATIDIS, Kyriakos, JIANG, Jing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Artificial Intelligence and Robotics
• Online Access: https://ink.library.smu.edu.sg/sis_research/8352; https://ink.library.smu.edu.sg/context/sis_research/article/9355/viewcontent/2023.findings_emnlp.683.pdf
• Institution: Singapore Management University

Humans possess a strong capability for reasoning beyond common sense. For example, given an unconventional image of a goldfish laying on the table next to an empty fishbowl, a human would effortlessly determine that the fish is not inside the fishbowl. The case, however, may be different for a vision-language model, whose reasoning could gravitate towards the common scenario that the fish is inside the bowl, despite the visual input. In this paper, we introduce a novel probing dataset named ROME (reasoning beyond commonsense knowledge) to evaluate whether the state-of-the-art pre-trained vision-language models have the reasoning capability to correctly interpret counter-intuitive content. ROME contains images that defy commonsense knowledge with regards to color, shape, material, size and positional relation. Experiments on the state-of-the-art pre-trained vision-language models reveal that most of these models are still largely incapable of interpreting counter-intuitive scenarios. We hope that ROME will spur further investigations on reasoning beyond commonsense knowledge in vision-language research.