Demonstrating multi-modal human instruction comprehension with AR smart glass

• Main Authors: WEERAKOON, Mudiyanselage Dulanga Kaveesha, SUBBARAJU, Vigneshwaran, TRAN, Tuan, MISRA, Archan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Human-AI Collaboration; Multi-Modal Networks; Pervasive Systems; Referring Expression Comprehension; Visual Grounding; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/7797; https://ink.library.smu.edu.sg/context/sis_research/article/8800/viewcontent/COMSNETS2023_Demo_CR.pdf
• Institution: Singapore Management University

We present a multi-modal human instruction comprehension prototype for object acquisition tasks that involve verbal, visual and pointing gesture cues. Our prototype includes an AR smart-glass for issuing the instructions and a Jetson TX2 pervasive device for executing comprehension algorithms. With...