Gesture-controlled semi-autonomous robotic arm with AHRS-based orientation tracking and real-time object detection

• Main Author: Loh, Aloysius Sijing
• Other Authors: Oh Hong Lye
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/181500
• Institution: Nanyang Technological University

This report presents the development of a semi-autonomous robotic arm system designed for dynamic environments, integrating gesture control, object detection, and robotic manipulation. The system employed an Adafruit Feather nRF52840 Sense microcontroller equipped with a 9- Degrees of Freedom inertial measurement unit, running an altitude and heading reference system (AHRS) algorithm to measure orientation for precise arm movement control. The robotic arm’s operations were further enhanced by a TensorFlow Lite gesture recognition model, optimised for low-latency inference on the microcontroller, and a YOLOv4-Tiny image recognition model running on a Raspberry Pi 4 for real-time object detection. Communication between components was achieved through Bluetooth Low Energy (BLE), ensuring seamless interaction and efficient data transfer. Multithreading on the Raspberry Pi ensured smooth operation by handling BLE communication, camera input, and object detection concurrently. This distributed architecture allowed the robotic arm to respond to user input while performing object detection in real time. This project demonstrates the potential of combining AHRS-based orientation tracking, gesture recognition, and object detection to enhance human-robot interaction in research, manufacturing, and hazardous environments.