Enhancing learning in virtual reality: utilizing hand motion detection in educational games

• Main Author: Phang, Siang Jian
• Other Authors: Cai Yiyu
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Science::General::Education; Engineering::Computer science and engineering::Software; Engineering::Mechanical engineering
• Online Access: https://hdl.handle.net/10356/167988
• Institution: Nanyang Technological University

A Virtual Reality (VR) game’s ability to provide immersive experiences is an extremely valuable property, especially in an educational setting, where players are often trained in scenarios that they are required to perform in real life. However, most existing VR educational games break the immersion through player interactions that are not true to their real-life counterpart, which limits the potential of VR games. To address this issue, a script that tracks and detects specific and customizable hand motion that the player performs in real-time is developed. This script allows game developers to implement realistic hand motions into portions of the game that focuses on the player’s hand motion which enhances the overall learning experience of players. Another script is also developed that is capable of showing a replay of the player’s hand motion performed during the game which can be used in the debriefing portion of the game. An experiment was conducted detailing the experiences of the subjects while playing an aircraft hydraulic maintenance game with the developed script implemented into the game. The script implementation in this game is set to detect a nut loosening motion that the player is tasked to perform. The survey results demonstrate that most players agree that the hand motion tracking and detection script enhances the interactivity and learning experiences of VR games. This technology has the potential to elevate the learning experience for various educational applications, such as medical and engineering training simulations, as well as entertainment-focused VR games.