Development of gesture control of robotic arm/robotic structures for assistive systems

• Main Author: Tan, Yi Ming
• Other Authors: Tay, Wee Peng
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2021
• Subjects: Engineering::Electrical and electronic engineering::Computer hardware, software and systems
• Online Access: https://hdl.handle.net/10356/150122
• Institution: Nanyang Technological University

The development and use of collaborative robots have been on the rise in recent years, enabling humans to perform tasks with precision and ability that would otherwise be humanly impossible. Many of such robotic applications and studies conducted were to simplify or automate work processes and reduce the workload of general, able-bodied users. Little research and work were done to investigate the possibilities of new interfaces which allow users with physical limitations to interact and control robotic systems. This project proposes a method for interaction making use of a Leap Motion Controller, an LSTM machine learning model, and a Niryo Ned robotic arm. The Leap Motion Controller, an optical hand tracking module was used to read and return data of hands and fingers detected. An LSTM model was trained to classify and recognize a set of 9 different gestures and the classified gestures are then used to control a robotic arm. The final trained LSTM model was able to achieve an F-score of 92% when evaluated against a test set of 9 gestures. After, it was successfully integrated and implemented with a Niryo Ned robotic arm which was tasked to perform 8 actions. The development of this system would be beneficial especially in the field of assistive technology, giving individuals with physical limitations and disabilities ways to overcome their limitations, enabling them to live independently in this world that is largely catered to the able-bodied majority.