Jointly optimizing sensing pipelines for multimodal mixed reality interaction

Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-cons...

Full description

Saved in:
Bibliographic Details
Main Authors: RATHNAYAKE, Darshana, DE SILVA, Ashen, PUWAKDANDAWA, Dasun, MEEGAHAPOLA, Lakmal, MISRA, Archan, PERERA, Indika
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2020
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/5669
https://ink.library.smu.edu.sg/context/sis_research/article/6671/viewcontent/24._Jointly_Optimizing_Sensing_Pipelines_for_Multimodal_Mixed_Reality_In....pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-6671
record_format dspace
spelling sg-smu-ink.sis_research-66712021-07-09T05:30:26Z Jointly optimizing sensing pipelines for multimodal mixed reality interaction RATHNAYAKE, Darshana DE SILVA, Ashen PUWAKDANDAWA, Dasun MEEGAHAPOLA, Lakmal MISRA, Archan PERERA, Indika Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency--vs.--accuracy tradeoff by exploiting cross-modal dependencies -- i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative "classroom" context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase 10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction. 2020-12-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/5669 info:doi/10.1109/MASS50613.2020.00046 https://ink.library.smu.edu.sg/context/sis_research/article/6671/viewcontent/24._Jointly_Optimizing_Sensing_Pipelines_for_Multimodal_Mixed_Reality_In....pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University sensor fusion mixed reality multimodal interactions Software Engineering
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic sensor fusion
mixed reality
multimodal interactions
Software Engineering
spellingShingle sensor fusion
mixed reality
multimodal interactions
Software Engineering
RATHNAYAKE, Darshana
DE SILVA, Ashen
PUWAKDANDAWA, Dasun
MEEGAHAPOLA, Lakmal
MISRA, Archan
PERERA, Indika
Jointly optimizing sensing pipelines for multimodal mixed reality interaction
description Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency--vs.--accuracy tradeoff by exploiting cross-modal dependencies -- i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative "classroom" context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase 10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction.
format text
author RATHNAYAKE, Darshana
DE SILVA, Ashen
PUWAKDANDAWA, Dasun
MEEGAHAPOLA, Lakmal
MISRA, Archan
PERERA, Indika
author_facet RATHNAYAKE, Darshana
DE SILVA, Ashen
PUWAKDANDAWA, Dasun
MEEGAHAPOLA, Lakmal
MISRA, Archan
PERERA, Indika
author_sort RATHNAYAKE, Darshana
title Jointly optimizing sensing pipelines for multimodal mixed reality interaction
title_short Jointly optimizing sensing pipelines for multimodal mixed reality interaction
title_full Jointly optimizing sensing pipelines for multimodal mixed reality interaction
title_fullStr Jointly optimizing sensing pipelines for multimodal mixed reality interaction
title_full_unstemmed Jointly optimizing sensing pipelines for multimodal mixed reality interaction
title_sort jointly optimizing sensing pipelines for multimodal mixed reality interaction
publisher Institutional Knowledge at Singapore Management University
publishDate 2020
url https://ink.library.smu.edu.sg/sis_research/5669
https://ink.library.smu.edu.sg/context/sis_research/article/6671/viewcontent/24._Jointly_Optimizing_Sensing_Pipelines_for_Multimodal_Mixed_Reality_In....pdf
_version_ 1770575558052151296