Gesture enhanced comprehension of ambiguous human-to-robot instructions
This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based...
Saved in:
Main Authors: | , , , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2020
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/5369 https://ink.library.smu.edu.sg/context/sis_research/article/6373/viewcontent/26._Gesture_Enhanced_Comprehension_of_Ambiguous_Human_To_Robot_Instructi....pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-6373 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-63732020-11-19T06:49:42Z Gesture enhanced comprehension of ambiguous human-to-robot instructions WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, SUBBARAJU, Vigneshwaran KARUMPULLI, Nipuni TRAN, Minh Anh Tuan XU, Qianli TAN, U-Xuan LIM, Joo Hwee MISRA, Archan This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based text parsing with a novel knowledge-graph traversal mechanism, over a multi-modal input of vision, natural language text and pointing. Via multiple studies related to a benchmark table top manipulation task, we show that (a) M2Gestic can achieve close-to-human performance in reasoning over unambiguous verbal instructions, and (b) incorporating pointing input (even with its inherent location uncertainty) in M2Gestic results in a significant (30%) accuracy improvement when verbal instructions are ambiguous. 2020-10-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/5369 info:doi/10.1145/3382507.3418863 https://ink.library.smu.edu.sg/context/sis_research/article/6373/viewcontent/26._Gesture_Enhanced_Comprehension_of_Ambiguous_Human_To_Robot_Instructi....pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
spellingShingle |
Artificial Intelligence and Robotics Graphics and Human Computer Interfaces WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, SUBBARAJU, Vigneshwaran KARUMPULLI, Nipuni TRAN, Minh Anh Tuan XU, Qianli TAN, U-Xuan LIM, Joo Hwee MISRA, Archan Gesture enhanced comprehension of ambiguous human-to-robot instructions |
description |
This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based text parsing with a novel knowledge-graph traversal mechanism, over a multi-modal input of vision, natural language text and pointing. Via multiple studies related to a benchmark table top manipulation task, we show that (a) M2Gestic can achieve close-to-human performance in reasoning over unambiguous verbal instructions, and (b) incorporating pointing input (even with its inherent location uncertainty) in M2Gestic results in a significant (30%) accuracy improvement when verbal instructions are ambiguous. |
format |
text |
author |
WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, SUBBARAJU, Vigneshwaran KARUMPULLI, Nipuni TRAN, Minh Anh Tuan XU, Qianli TAN, U-Xuan LIM, Joo Hwee MISRA, Archan |
author_facet |
WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, SUBBARAJU, Vigneshwaran KARUMPULLI, Nipuni TRAN, Minh Anh Tuan XU, Qianli TAN, U-Xuan LIM, Joo Hwee MISRA, Archan |
author_sort |
WEERAKOON MUDIYANSELAGE DULANGA KAVEESHA WEERAKOON, |
title |
Gesture enhanced comprehension of ambiguous human-to-robot instructions |
title_short |
Gesture enhanced comprehension of ambiguous human-to-robot instructions |
title_full |
Gesture enhanced comprehension of ambiguous human-to-robot instructions |
title_fullStr |
Gesture enhanced comprehension of ambiguous human-to-robot instructions |
title_full_unstemmed |
Gesture enhanced comprehension of ambiguous human-to-robot instructions |
title_sort |
gesture enhanced comprehension of ambiguous human-to-robot instructions |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2020 |
url |
https://ink.library.smu.edu.sg/sis_research/5369 https://ink.library.smu.edu.sg/context/sis_research/article/6373/viewcontent/26._Gesture_Enhanced_Comprehension_of_Ambiguous_Human_To_Robot_Instructi....pdf |
_version_ |
1770575435436916736 |