Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors

• Main Authors: Wang, Ming, Yan, Zheng, Wang, Ting, Cai, Pingqiang, Gao, Siyu, Zeng, Yi, Wan, Changjin, Wang, Hong, Pan, Liang, Yu, Jiancan, Pan, Shaowu, He, Ke, Lu, Jie, Chen, Xiaodong
• Other Authors: School of Materials Science and Engineering
• Format: Article
• Language: English
• Published: 2021
• Subjects: Engineering::Materials; Convolutional Neural Networks; Network architecture
• Online Access: https://hdl.handle.net/10356/148021
• Institution: Nanyang Technological University

Gesture recognition using machine learning methods is valuable in the development of advanced cybernetics, robotics, and healthcare systems, and typically relies on images or videos. To improve recognition accuracy, such visual data can be fused with data from other sensors, but this approach is limited by the quality of the sensor data and the incompatibility of the datasets. Here, we report a bioinspired data fusion architecture that can perform human gesture recognition by integrating visual data with somatosensory data from skin-like stretchable strain sensors. The learning architecture uses a convolutional neural network for visual processing, and then implements a sparse neural network for sensor data fusion and recognition. Our approach can achieve a recognition accuracy of 100%, and maintain recognition accuracy with noisy, under- or over-exposed images. We also show that our architecture can be implemented for robot navigation using hand gestures with a small error, even in the dark.