Smartwatch-based early gesture detection & trajectory tracking for interactive gesture-driven applications

• Main Authors: VU, Tran Huy, MISRA, Archan, ROY, Quentin, CHOO, Kenny Tsu Wei, LEE, Youngki
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2018
• Subjects: VR; gesture recognition; hand tracking; immersive applications; low-latency; wearable devices; Information Security; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/4253; https://ink.library.smu.edu.sg/context/sis_research/article/5256/viewcontent/ubicomp18_tabletennis_vu.pdf
• Institution: Singapore Management University

The paper explores the possibility of using wrist-worn devices (specifically, a smartwatch) to accurately track the hand movement and gestures for a new class of immersive, interactive gesture-driven applications. These interactive applications need two special features: (a) the ability to identify gestures from a continuous stream of sensor data early–i.e., even before the gesture is complete, and (b) the ability to precisely track the hand’s trajectory, even though the underlying inertial sensor data is noisy. We develop a new approach that tackles these requirements by first building a HMM-based gesture recognition framework that does not need an explicit segmentation step, and then using a per-gesture trajectory tracking solution that tracks the hand movement only during these predefined gestures. Using an elaborate setup that allows us to realistically study the table-tennis related hand movements of users, we show that our approach works: (a) it can achieve 95% stroke recognition accuracy. Within 50% of gesture, it can achieve a recall value of 92% for 10 novice users and 93% for 15 experienced users from a continuous sensor stream; (b) it can track hand movement during such stroke play with a median accuracy of 6.2 cm