A unified 3D human motion synthesis model via conditional variational auto-encoder

• Main Authors: Cai, Yujun, Wang, Yiwei, Zhu, Yiheng, Cham, Tat-Jen, Cai, Jianfei, Yuan, Junsong, Liu, Jun, Zheng, Chuanxia, Yan, Sijie, Ding, Henghui, Shen, Xiaohui, Liu, Ding, Thalmann, Nadia Magnenat
• Other Authors: School of Computer Science and Engineering
• Format: Conference or Workshop Item
• Language: English
• Published: 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision; Gestures and Body Pose; Image and Video Synthesis
• Online Access: https://hdl.handle.net/10356/172651
• Institution: Nanyang Technological University

We present a unified and flexible framework to address the generalized problem of 3D motion synthesis that covers the tasks of motion prediction, completion, interpolation, and spatial-temporal recovery. Since these tasks have different input constraints and various fidelity and diversity requirements, most existing approaches only cater to a specific task or use different architectures to address various tasks. Here we propose a unified framework based on Conditional Variational Auto-Encoder (CVAE), where we treat any arbitrary input as a masked motion series. Notably, by considering this problem as a conditional generation process, we estimate a parametric distribution of the missing regions based on the input conditions, from which to sample and synthesize the full motion series. To further allow the flexibility of manipulating the motion style of the generated series, we design an Action-Adaptive Modulation (AAM) to propagate the given semantic guidance through the whole sequence. We also introduce a cross-attention mechanism to exploit distant relations among decoder and encoder features for better realism and global consistency. We conducted extensive experiments on Human 3.6M and CMU-Mocap. The results show that our method produces coherent and realistic results for various motion synthesis tasks, with the synthesized motions distinctly adapted by the given action labels.