Emergent semantic segmentation: training-free dense-label-free extraction from vision-language models

• Main Author: Luo, Jiayun
• Other Authors: Li Boyang
• Format: Thesis-Master by Research
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Vision-language model; Open-vocabulary semantic segementation
• Online Access: https://hdl.handle.net/10356/175765
• Institution: Nanyang Technological University

From an enormous amount of image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which is vital for tasks such as image captioning and visual question answering. However, leveraging such pre-trained models for open-vocabulary semantic segmentation remains a challenge. In this thesis, we propose a simple, yet extremely effective, training-free technique, Plug-and-Play Open-Vocabulary Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with direct text-to-image cross-attention and an image-text matching loss to produce semantic segmentation. However, cross-attention alone tends to over-segment, whereas cross-attention plus GradCAM tend to under-segment. To alleviate this issue, we introduce Salience Dropout; by iteratively dropping patches that the model is most attentive to, we are able to better resolve the entire extent of the segmentation mask. PnP-OVSS does not require any neural network training and performs hyperparameter tuning without the need for any segmentation annotations, even for a validation set. PnP-OVSS demonstrates substantial improvements over comparable baselines (+29.4\% on PASCAL VOC, +13.2\% on PASCAL Context, +14.0\% mIoU on MS COCO, +2.4\% on COCO Stuff) and even outperforms most baselines that conduct additional network training on top of pretrained VLMs.