Weakly paired multi-domain image translation

• Main Authors: ZHANG, M.Y., HUANG, Zhiwu, PAUDEL, D.P., THOMA, J., VAN, Gool L.
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/6412; https://ink.library.smu.edu.sg/context/sis_research/article/7415/viewcontent/Weakly_Paired_Multi_Domain_Image.pdf
• Institution: Singapore Management University

In this paper, we aim at studying the new problem of weakly paired multi-domain image translation. To this end, we collect a dataset that contains weakly paired images from multiple domains. Two images are considered to be weakly paired if they are captured from nearby locations and share an overlapping field of view. These images are possibly captured by two asynchronous cameras—often resulting in images from separate domains, e.g. summer and winter. Major motivations for using weakly paired images are: (i) performance improvement towards that of paired data; (ii) cheap labels and abundant data availability. For the first time in this paper, we propose a multi-domain image translation method specifically designed for weakly paired data. The proposed method consists of an attention-based generator and a two-stream discriminator that deals with misalignment between source and target images. Our method generates images in the target domain while preserving source image content, including foreground objects such as cars and pedestrians. Our extensive experiments demonstrate the superiority of the proposed method in comparison to the state-of-the-art.