Demo abstract: VGGlass - Demonstrating visual grounding and localization synergy with a LiDAR-enabled smart-glass

• Main Authors: RATHNAYAKE, Darshana, WEERAKOON, Dulanga, RADHAKRISHNAN, Meeralakshmi, SUBBARAJU, Vigneshwaran, HWANG, Inseok, MISRA, Archan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Multi-modal interaction; 3D Localization; Visual Grounding; Computer Engineering; Graphics and Human Computer Interfaces; OS and Networks
• Online Access: https://ink.library.smu.edu.sg/sis_research/9221; https://ink.library.smu.edu.sg/context/sis_research/article/10176/viewcontent/Sensys2023_VGGlass_DemoPaper_CamReady.pdf
• Institution: Singapore Management University

This work demonstrates the VGGlass system, which simultaneously interprets human instructions for a target acquisition task and determines the precise 3D positions of both user and the target object. This is achieved by utilizing LiDARs mounted in the infrastructure and a smart glass device worn by the user. Key to our system is the union of LiDAR-based localization termed LiLOC and a multi-modal visual grounding approach termed RealG(2)In-Lite. To demonstrate the system, we use Intel RealSense L515 cameras and a Microsoft HoloLens 2, as the user devices. VGGlass is able to: a) track the user in real-time in a global coordinate system, and b) locate target objects referred by natural language and pointing gestures.