Gesture enhanced comprehension of ambiguous human-to-robot instructions

This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based...

Full description

Saved in:
Bibliographic Details
Main Authors: WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, SUBBARAJU, Vigneshwaran, KARUMPULLI, Nipuni, TRAN, Minh Anh Tuan, XU, Qianli, TAN, U-Xuan, LIM, Joo Hwee, MISRA, Archan
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2020
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/5369
https://ink.library.smu.edu.sg/context/sis_research/article/6373/viewcontent/26._Gesture_Enhanced_Comprehension_of_Ambiguous_Human_To_Robot_Instructi....pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
Description
Summary:This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based text parsing with a novel knowledge-graph traversal mechanism, over a multi-modal input of vision, natural language text and pointing. Via multiple studies related to a benchmark table top manipulation task, we show that (a) M2Gestic can achieve close-to-human performance in reasoning over unambiguous verbal instructions, and (b) incorporating pointing input (even with its inherent location uncertainty) in M2Gestic results in a significant (30%) accuracy improvement when verbal instructions are ambiguous.