Learning an interpretable stylized subspace for 3D-aware animatable artforms

• Main Authors: ZHENG, Chenxi, LIU, Bangzhen, XU, Xuemiao, ZHANG, Huaidong, HE, Shengfeng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: 3D-aware GANs; Adaptation models; facial attribute editing; Image reconstruction; Painting; stylized animation; Three-dimensional displays; Training; Graphics and Human Computer Interfaces; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/8697; https://ink.library.smu.edu.sg/context/sis_research/article/9700/viewcontent/LearningInterpretableStylizedAnimatable_Artforms_av.pdf
• Institution: Singapore Management University

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the pose and content of a given art painting. This anchor serves as a reliable indicator of the original latent space local structure, therefore sharing the same editable predefined expression vectors. In the second stage, we train a customized 3D-aware GAN specific to the input artform, while enforcing the preservation of the original latent local structure through a meticulous style-directional difference loss. This approach ensures the creation of a stylized sub-space that remains interpretable and retains 3D control. The effectiveness and versatility of 3DArtmator are validated through extensive experiments across a diverse range of art styles. With the ability to generate 3D reconstruction and editing for artforms while maintaining interpretability, 3DArtmator opens up new possibilities for artistic exploration and engagement.