Zero-shot out-of-distribution detection with outlier label exposure
As vision-language models like CLIP are widely applied to zero-shot tasks and gain remarkable performance on in-distribution (ID) data, detecting and rejecting out-of-distribution (OOD) inputs in the zero-shot setting have become crucial for ensuring the safety of using such models on the fly. Most...
Saved in:
Main Authors: | , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/9789 https://ink.library.smu.edu.sg/context/sis_research/article/10789/viewcontent/2406.01170v1.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-10789 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-107892024-12-16T01:51:53Z Zero-shot out-of-distribution detection with outlier label exposure DING, Choubo PANG, Guansong As vision-language models like CLIP are widely applied to zero-shot tasks and gain remarkable performance on in-distribution (ID) data, detecting and rejecting out-of-distribution (OOD) inputs in the zero-shot setting have become crucial for ensuring the safety of using such models on the fly. Most existing zero-shot OOD detectors rely on ID class label-based prompts to guide CLIP in classifying ID images and rejecting OOD images. In this work we instead propose to leverage a large set of diverse auxiliary outlier class labels as pseudo OOD class text prompts to CLIP for enhancing zero-shot OOD detection, an approach we called Outlier Label Exposure (OLE). The key intuition is that ID images are expected to have lower similarity to these outlier class prompts than OOD images. One issue is that raw class labels often include noise labels, e.g., synonyms of ID labels, rendering raw OLE-based detection ineffective. To address this issue, we introduce an outlier prototype learning module that utilizes the prompt embeddings of the outlier labels to learn a small set of pivotal outlier prototypes for an embedding similarity-based OOD scoring. Additionally, the outlier classes and their prototypes can be loosely coupled with the ID classes, leading to an inseparable decision region between them. Thus, we also introduce an outlier label generation module that synthesizes our outlier prototypes and ID class embeddings to generate in-between outlier prototypes to further calibrate the detection in OLE. Despite its simplicity, extensive experiments show that OLE substantially improves detection performance and achieves new state-of-the-art performance in large-scale OOD and hard OOD detection benchmarks. 2024-06-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9789 info:doi/10.1109/IJCNN60899.2024.10650173 https://ink.library.smu.edu.sg/context/sis_research/article/10789/viewcontent/2406.01170v1.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Out-of-distribution detection Zero-shot detection Prompt engineering Artificial Intelligence and Robotics Databases and Information Systems |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Out-of-distribution detection Zero-shot detection Prompt engineering Artificial Intelligence and Robotics Databases and Information Systems |
spellingShingle |
Out-of-distribution detection Zero-shot detection Prompt engineering Artificial Intelligence and Robotics Databases and Information Systems DING, Choubo PANG, Guansong Zero-shot out-of-distribution detection with outlier label exposure |
description |
As vision-language models like CLIP are widely applied to zero-shot tasks and gain remarkable performance on in-distribution (ID) data, detecting and rejecting out-of-distribution (OOD) inputs in the zero-shot setting have become crucial for ensuring the safety of using such models on the fly. Most existing zero-shot OOD detectors rely on ID class label-based prompts to guide CLIP in classifying ID images and rejecting OOD images. In this work we instead propose to leverage a large set of diverse auxiliary outlier class labels as pseudo OOD class text prompts to CLIP for enhancing zero-shot OOD detection, an approach we called Outlier Label Exposure (OLE). The key intuition is that ID images are expected to have lower similarity to these outlier class prompts than OOD images. One issue is that raw class labels often include noise labels, e.g., synonyms of ID labels, rendering raw OLE-based detection ineffective. To address this issue, we introduce an outlier prototype learning module that utilizes the prompt embeddings of the outlier labels to learn a small set of pivotal outlier prototypes for an embedding similarity-based OOD scoring. Additionally, the outlier classes and their prototypes can be loosely coupled with the ID classes, leading to an inseparable decision region between them. Thus, we also introduce an outlier label generation module that synthesizes our outlier prototypes and ID class embeddings to generate in-between outlier prototypes to further calibrate the detection in OLE. Despite its simplicity, extensive experiments show that OLE substantially improves detection performance and achieves new state-of-the-art performance in large-scale OOD and hard OOD detection benchmarks. |
format |
text |
author |
DING, Choubo PANG, Guansong |
author_facet |
DING, Choubo PANG, Guansong |
author_sort |
DING, Choubo |
title |
Zero-shot out-of-distribution detection with outlier label exposure |
title_short |
Zero-shot out-of-distribution detection with outlier label exposure |
title_full |
Zero-shot out-of-distribution detection with outlier label exposure |
title_fullStr |
Zero-shot out-of-distribution detection with outlier label exposure |
title_full_unstemmed |
Zero-shot out-of-distribution detection with outlier label exposure |
title_sort |
zero-shot out-of-distribution detection with outlier label exposure |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/9789 https://ink.library.smu.edu.sg/context/sis_research/article/10789/viewcontent/2406.01170v1.pdf |
_version_ |
1819113139384352768 |