Transferring and regularizing prediction for semantic segmentation
Semantic segmentation often requires a large set of images with pixel-level annotations. In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e.g., computer games) with computer-generated annotations can be adapted t...
Saved in:
Main Authors: | , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2020
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/6474 https://ink.library.smu.edu.sg/context/sis_research/article/7477/viewcontent/Zhang_Transferring_and_Regularizing_Prediction_for_Semantic_Segmentation_CVPR_2020_paper.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-7477 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-74772022-01-10T05:40:30Z Transferring and regularizing prediction for semantic segmentation ZHANG, Yiheng QIU, Zhaofan YAO, Ting NGO, Chong-wah LIU, Dong MEI, Tao Semantic segmentation often requires a large set of images with pixel-level annotations. In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e.g., computer games) with computer-generated annotations can be adapted to real images. Despite this progress, without constraining the prediction on real images, the models will easily overfit on synthetic data due to severe domain mismatch. In this paper, we novelly exploit the intrinsic properties of semantic segmentation to alleviate such problem for model transfer. Specifically, we present a Regularizer of Prediction Transfer (RPT) that imposes the intrinsic properties as constraints to regularize model transfer in an unsupervised fashion. These constraints include patch-level, cluster-level and context-level semantic prediction consistencies at different levels of image formation. As the transfer is label-free and data-driven, the robustness of prediction is addressed by selectively involving a subset of image regions for model regularization. Extensive experiments are conducted to verify the proposal of RPT on the transfer of models trained on GTA5 and SYNTHIA (synthetic data) to Cityscapes dataset (urban street scenes). RPT shows consistent improvements when injecting the constraints on several neural networks for semantic segmentation. More remarkably, when integrating RPT into the adversarial-based segmentation framework, we report to-date the best results: mIoU of 53.2%/51.7% when transferring from GTA5/SYNTHIA to Cityscapes, respectively. 2020-06-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/6474 info:doi/10.1109/CVPR42600.2020.00964 https://ink.library.smu.edu.sg/context/sis_research/article/7477/viewcontent/Zhang_Transferring_and_Regularizing_Prediction_for_Semantic_Segmentation_CVPR_2020_paper.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Data Storage Systems Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Data Storage Systems Graphics and Human Computer Interfaces |
spellingShingle |
Data Storage Systems Graphics and Human Computer Interfaces ZHANG, Yiheng QIU, Zhaofan YAO, Ting NGO, Chong-wah LIU, Dong MEI, Tao Transferring and regularizing prediction for semantic segmentation |
description |
Semantic segmentation often requires a large set of images with pixel-level annotations. In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e.g., computer games) with computer-generated annotations can be adapted to real images. Despite this progress, without constraining the prediction on real images, the models will easily overfit on synthetic data due to severe domain mismatch. In this paper, we novelly exploit the intrinsic properties of semantic segmentation to alleviate such problem for model transfer. Specifically, we present a Regularizer of Prediction Transfer (RPT) that imposes the intrinsic properties as constraints to regularize model transfer in an unsupervised fashion. These constraints include patch-level, cluster-level and context-level semantic prediction consistencies at different levels of image formation. As the transfer is label-free and data-driven, the robustness of prediction is addressed by selectively involving a subset of image regions for model regularization. Extensive experiments are conducted to verify the proposal of RPT on the transfer of models trained on GTA5 and SYNTHIA (synthetic data) to Cityscapes dataset (urban street scenes). RPT shows consistent improvements when injecting the constraints on several neural networks for semantic segmentation. More remarkably, when integrating RPT into the adversarial-based segmentation framework, we report to-date the best results: mIoU of 53.2%/51.7% when transferring from GTA5/SYNTHIA to Cityscapes, respectively. |
format |
text |
author |
ZHANG, Yiheng QIU, Zhaofan YAO, Ting NGO, Chong-wah LIU, Dong MEI, Tao |
author_facet |
ZHANG, Yiheng QIU, Zhaofan YAO, Ting NGO, Chong-wah LIU, Dong MEI, Tao |
author_sort |
ZHANG, Yiheng |
title |
Transferring and regularizing prediction for semantic segmentation |
title_short |
Transferring and regularizing prediction for semantic segmentation |
title_full |
Transferring and regularizing prediction for semantic segmentation |
title_fullStr |
Transferring and regularizing prediction for semantic segmentation |
title_full_unstemmed |
Transferring and regularizing prediction for semantic segmentation |
title_sort |
transferring and regularizing prediction for semantic segmentation |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2020 |
url |
https://ink.library.smu.edu.sg/sis_research/6474 https://ink.library.smu.edu.sg/context/sis_research/article/7477/viewcontent/Zhang_Transferring_and_Regularizing_Prediction_for_Semantic_Segmentation_CVPR_2020_paper.pdf |
_version_ |
1770575970921611264 |