Body movement mimic – video-based human body motion transfer

• Main Author: Mamuduri, Paulani
• Other Authors: Cham Tat Jen
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: https://hdl.handle.net/10356/166084
• Institution: Nanyang Technological University

In the age of rapid technological advancement, we have seen deepfake and even fallen for them. While many of the famous deepfake videos are for laughs and entertainment, such technology can significantly transform many areas of society today. In this report, we focus on the human body motion transfer, whereby the motion of a body in a driving video is transferred onto a body in a 2D image. We have studied current technology, First Order Motion Model, which performs well for faces but not human body motion. We have proposed two approaches to improve model performance on the human body. The first approach used depth maps from pre-trained depth estimator MiDaS to guide keypoint detection in the First Order Motion Model. It did drastically help improve motion transfer, given that the quality of depth maps is good. The second approach trained the model with different keypoints to find an optimal one, given the nature of the dataset. Results show that the nature of the datasets affects the model performances.