Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models
Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream...
Saved in:
Main Authors: | , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2023
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8473 https://ink.library.smu.edu.sg/context/sis_research/article/9476/viewcontent/Generalized_Logit_Adjustment__Calibrating_Fine_tuned_Models_by_Removing_Label_Bias_in_Foundation_Models__1_.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9476 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-94762024-01-04T09:26:05Z Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models ZHU, Beier TANG, Kaihua SUN, Qianru ZHANG, Hanwang Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like in traditional long-tailed classification tasks. To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, a large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. 2023-12-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8473 https://ink.library.smu.edu.sg/context/sis_research/article/9476/viewcontent/Generalized_Logit_Adjustment__Calibrating_Fine_tuned_Models_by_Removing_Label_Bias_in_Foundation_Models__1_.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Databases and Information Systems |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Databases and Information Systems |
spellingShingle |
Databases and Information Systems ZHU, Beier TANG, Kaihua SUN, Qianru ZHANG, Hanwang Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
description |
Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like in traditional long-tailed classification tasks. To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, a large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. |
format |
text |
author |
ZHU, Beier TANG, Kaihua SUN, Qianru ZHANG, Hanwang |
author_facet |
ZHU, Beier TANG, Kaihua SUN, Qianru ZHANG, Hanwang |
author_sort |
ZHU, Beier |
title |
Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
title_short |
Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
title_full |
Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
title_fullStr |
Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
title_full_unstemmed |
Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models |
title_sort |
generalized logit adjustment: calibrating fine-tuned models by removing label bias in foundation models |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2023 |
url |
https://ink.library.smu.edu.sg/sis_research/8473 https://ink.library.smu.edu.sg/context/sis_research/article/9476/viewcontent/Generalized_Logit_Adjustment__Calibrating_Fine_tuned_Models_by_Removing_Label_Bias_in_Foundation_Models__1_.pdf |
_version_ |
1787590775997988864 |