Non-verbal information estimation in multi-party human-robot/virtual human interaction

• Main Author: Zhang, Zhijie
• Other Authors: Zheng Jianmin
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence; Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision; Social sciences::Psychology::Affection and emotion
• Online Access: https://hdl.handle.net/10356/168493
• Institution: Nanyang Technological University

Robots and virtual agents have been deployed in various fields, and are playing an increasingly important role in human being’s daily lives. Thus, these intelligent agents (IAs) are required to interact with their users appropriately. To achieve this goal, IAs need to understand human social signals, before generating socially acceptable responses. However, current multi-party social human-robot interaction (HRI) is still far from being satisfactory. Unlike dyadic HRI, multi-party HRI involves more than one participant in the interaction, so with the increase in the participant number, IAs face more challenging tasks. The overall objective of this research is to investigate and develop new techniques to empower robots or virtual agents with the ability to understand the behaviors, intentions, and affects of the participants in multi-party social interaction, which helps the agent manage multi-party issues in social HRI. Specifically, this thesis presents new methods to analyze and estimate four types of non-verbal social information in multi-party human-robot interaction scenarios, namely (i) engagement intention estimation, (ii) engagement estimation during interaction, (iii) personality estimation, and (iv) emotion recognition.