Vision language representation learning - Yang, Xiaofeng (Nanyang Technological University, 2023)
Vision and language tasks have garnered increasing research attention in recent years, finding extensive applications in commercial software such as Microsoft Office, Amazon cloud services, and the iOS photo app. Vision language representation learning is a crucial component of numerous vision language tasks, including visual question answering (VQA), image captioning, visual reasoning, and visual navigation. Despite its significance, learning effective vision language representation remains challenging due to several reasons. Firstly, visual information and language information operate at different levels. Visual information, like image pixels, typically constitutes low-level information, while language information generally carries high-level information with semantic meanings. Thus, establishing suitable correspondences between visual and language concepts proves arduous. Secondly, language information encompasses both objects and object relations, whereas extracting and representing relations among visual concepts pose difficulties. Thirdly, vision and language tasks are often trained and evaluated on biased medium-sized datasets, which limits the ability to learn general representations exclusively from such datasets. Finally, vision language models' predictions lack explainability.