Transformer based gesture detection and control for robotic arm

• Main Author: Fu, Shihao
• Other Authors: Cheah Chien Chern
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Electrical and electronic engineering::Control and instrumentation::Robotics; Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: https://hdl.handle.net/10356/167277
• Institution: Nanyang Technological University

Transformer is the state-of-the-art neural network which implements attention mechanism. Many transformer-based models have been developed in recent years. Moreover, Shifted Window (SWIN) Transformer is a novel transformer-based model that was recently proposed for visual recognition tasks such as object detection and semantic segmentation. It is designed to process input images in a multi-scale and multi-resolution fashion, which allows it to process large images efficiently without requiring excessive amounts of memory or compute power. This paper developed a real-time gesture control method for the movement of UR5e robotic arm. A SWIN transformer model was fine-tuned using the collected data to detect open hand and fist. The model was then tested and deployed to be used by the robot. It was shown that the SWIN transformer performed accurately, and the gesture control system was able to perform tasks of moving, grabbing and placing successfully.