Construction of human digital twin model based on multimodal data and its application in locomotion mode identification

• Main Authors: Zhong, Ruirui, Hu, Bingtao, Feng, Yixiong, Zheng, Hao, Hong, Zhaoxi, Lou, Shanhe, Tan, Jianrong
• Other Authors: School of Mechanical and Aerospace Engineering
• Format: Article
• Language: English
• Published: 2024
• Subjects: Engineering; Human digital twin; Human-cyber-physical system
• Online Access: https://hdl.handle.net/10356/173796
• Institution: Nanyang Technological University

With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efficient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difficult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identification is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identification models. The experimental results proved the superiority of the HDT framework for human locomotion mode identification.