Hand Gesture Recognition for Smartphone-based Augmented Reality

• Main Author: Vidal, Jr., Eric Cesar
• Format: text
• Published: Archīum Ateneo 2020
• Subjects: n/a
• Online Access: https://archium.ateneo.edu/theses-dissertations/437
• Institution: Ateneo De Manila University

Hand Gesture Recognition (HGR) is a principal input method in Augmented Reality (AR) applications for head-mounted displays (HMDs). The high cost and limited availability of HMDs led to the use of smartphones as an alternative AR consumption device, but contemporary smartphone hardware were not designed with HGR in mind, leading to an inferior AR experience. This study explored the development of a software-based framework implementing HGR as a principal input method for smartphone AR applications. The framework additionally facilitates the development of cross-platform AR applications for both HMD and smartphone configurations. During the course of this study, two enhanced algorithms were developed for the segmentation and feature-detection stages of HGR, respectively, in order to facilitate the detection of four user-interface hand gestures (pointing, air-tapping/grabbing, whole-hand positioning, and grasping). The results of a user experiment show that, despite the smartphone’s hardware limitations, the smartphone system is able to achieve a preliminary 95.83% user success rate and exhibits mostly minor usability differences compared to a Microsoft HoloLens system, with both systems running an identical proof-of-concept AR application written using our framework. Future work recommends addressing the lone major usability difference related to the limited field-of-view of the environment-facing cameras of the smartphone, as well as improving gesture detection accuracy, supporting more gestures, and developing full-fledged applications based on this framework.