Spatially-invariant style-codes controlled makeup transfer

• Main Authors: DENG, Han, HAN, Chu, CAI, Hongmin, HAN, Guoqiang, HE, Shengfeng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Break down; Component wise; Features extraction; Reference image; Spatial informations; Spatial misalignments; Spatially invariants; Three-component; Transfer problems; Two-step extraction; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/8526; https://ink.library.smu.edu.sg/context/sis_research/article/9529/viewcontent/Spatially_Invariant_Style_Codes_Controlled_Makeup_Transfer.pdf
• Institution: Singapore Management University

Transferring makeup from the misaligned reference image is challenging. Previous methods overcome this barrier by computing pixel-wise correspondences between two images, which is inaccurate and computational-expensive. In this paper, we take a different perspective to break down the makeup transfer problem into a two-step extraction-assignment process. To this end, we propose a Style-based Controllable GAN model that consists of three components, each of which corresponds to target style-code encoding, face identity features extraction, and makeup fusion, respectively. In particular, a Part-specific Style Encoder encodes the component-wise makeup style of the reference image into a style-code in an intermediate latent space W. The style-code discards spatial information and therefore is invariant to spatial misalignment. On the other hand, the style-code embeds component-wise information, enabling flexible partial makeup editing from multiple references. This style-code, together with source identity features, is integrated into a Makeup Fusion Decoder equipped with multiple AdaIN layers to generate the final result. Our proposed method demonstrates great flexibility on makeup transfer by supporting makeup removal, shade-controllable makeup transfer, and part-specific makeup transfer, even with large spatial misalignment. Extensive experiments demonstrate the superiority of our approach over state-of-the-art methods.