Extending Smartphone-Based Hand Gesture Recognition for Augmented Reality Applications with Two-Finger-Pinch and Thumb-Orientation Gestures

• Main Authors: Vidal, Jr., Eric Cesar E, Rodrigo, Ma. Mercedes T
• Format: text
• Published: Archīum Ateneo 2022
• Subjects: augmented reality; hand gesture recognition; image processing; usability; Computer Sciences; Physical Sciences and Mathematics
• Online Access: https://archium.ateneo.edu/discs-faculty-pubs/356; https://doi.org/10.1007/978-3-031-21707-4_15
• Institution: Ateneo De Manila University

Augmented Reality (AR) on smartphones, while limited compared to their headset-styled counterparts (e.g., Microsoft HoloLens and Magic Leap), may be improved by supporting Hand Gesture Recognition (HGR) without requiring the expensive hardware found on the said headsets. This study improves the overall smartphone support of an existing HGR framework, named Augmented-reality Mobile Device Gestures (AMDG), and adds two new gestures: two-finger-pinch and thumb-orientation gestures, which are alternatives to existing gestures that may help improve overall user experience. A user study is also conducted to evaluate the usability of these two gestures in a realistic scenario. This study finds that the two-finger-pinch gesture is possibly more usable than the air-tap gesture (popularized by HoloLens) for the purpose of moving AR objects in a scene, and that the thumb-orientation gesture is a potentially viable simplification of whole-hand orientation (with some caveats relating to ease of execution and detection).