POCE: pose-controllable expression editing

• Main Authors: Wu, Rongliang, Yu, Yingchen, Zhan, Fangneng, Zhang, Jiahui, Liao, Shengcai, Lu, Shijian
• Other Authors: School of Computer Science and Engineering
• Format: Article
• Language: English
• Published: 2024
• Subjects: Computer and Information Science; Facial Expression Editing; Image Synthesis
• Online Access: https://hdl.handle.net/10356/173502
• Institution: Nanyang Technological University

Facial expression editing has attracted increasing attention with the advance of deep neural networks in recent years. However, most existing methods suffer from compromised editing fidelity and limited usability as they either ignore pose variations (unrealistic editing) or require paired training data (not easy to collect) for pose controls. This paper presents POCE, an innovative pose-controllable expression editing network that can generate realistic facial expressions and head poses simultaneously with just unpaired training images. POCE achieves the more accessible and realistic pose-controllable expression editing by mapping face images into UV space, where facial expressions and head poses can be disentangled and edited separately. POCE has two novel designs. The first is self-supervised UV completion that allows to complete UV maps sampled under different head poses, which often suffer from self-occlusions and missing facial texture. The second is weakly-supervised UV editing that allows to generate new facial expressions with minimal modification of facial identity, where the synthesized expression could be controlled by either an expression label or directly transplanted from a reference UV map via feature transfer. Extensive experiments show that POCE can learn from unpaired face images effectively, and the learned model can generate realistic and high-fidelity facial expressions under various new poses.