Wearables for in-situ monitoring of cognitive states: Challenges and opportunities

• Main Authors: RADHAKRISHNAN, Meera, KANDAPPU, Thivya, GULATI, Manoj, MISRA, Archan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Earables; Multi-device sensing; Cognitive states; Acoustic sensing; Electromyography; Pressure; Data Science; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/7885; https://ink.library.smu.edu.sg/context/sis_research/article/8893/viewcontent/WristSense_2023_CameraReady.pdf
• Institution: Singapore Management University

We propose using wrist and ear-based sensing, via multiple novel and complementary modalities, to unobtrusively infer activity-aware, complex cognitive and affective states (such as confusion, boredom, and recall failure) of individuals. While state-of-the-art wearable devices are predominantly used (a) independently, with limited coordination among multiple devices, and (b) to capture macro-level physical activity and physiological state, we seek to expand the ambit of unobtrusive wearable sensing to capture the cognitive states while performing commonplace physical activities. Such states typically manifest via fine-grained, almost unobservable, microscopic head, face, and eye movements. We identify some of these fine-grained physical markers that serve as proxies for cognitive/affective states and show that earable-mounted pressure, EMG, and ultrasonic sensing hold promise for canturing such markers.