EmpathyEar : An open-source avatar multimodal empathetic chatbot
This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, Empa...
Saved in:
Main Authors: | , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/9695 https://ink.library.smu.edu.sg/context/sis_research/article/10695/viewcontent/2024.acl_demos.7.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-10695 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-106952024-11-28T09:05:31Z EmpathyEar : An open-source avatar multimodal empathetic chatbot FEI, Hao ZHANG, Han WANG, Bin LIAO, Lizi LIU, Qian CAMBRIA, Erik This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access. 2024-08-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9695 info:doi/10.18653/v1/2024.acl-demos.7 https://ink.library.smu.edu.sg/context/sis_research/article/10695/viewcontent/2024.acl_demos.7.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Multimodal chatbot Empathetic response generation Large language model LLMs Digital avatars Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Multimodal chatbot Empathetic response generation Large language model LLMs Digital avatars Artificial Intelligence and Robotics Graphics and Human Computer Interfaces |
spellingShingle |
Multimodal chatbot Empathetic response generation Large language model LLMs Digital avatars Artificial Intelligence and Robotics Graphics and Human Computer Interfaces FEI, Hao ZHANG, Han WANG, Bin LIAO, Lizi LIU, Qian CAMBRIA, Erik EmpathyEar : An open-source avatar multimodal empathetic chatbot |
description |
This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access. |
format |
text |
author |
FEI, Hao ZHANG, Han WANG, Bin LIAO, Lizi LIU, Qian CAMBRIA, Erik |
author_facet |
FEI, Hao ZHANG, Han WANG, Bin LIAO, Lizi LIU, Qian CAMBRIA, Erik |
author_sort |
FEI, Hao |
title |
EmpathyEar : An open-source avatar multimodal empathetic chatbot |
title_short |
EmpathyEar : An open-source avatar multimodal empathetic chatbot |
title_full |
EmpathyEar : An open-source avatar multimodal empathetic chatbot |
title_fullStr |
EmpathyEar : An open-source avatar multimodal empathetic chatbot |
title_full_unstemmed |
EmpathyEar : An open-source avatar multimodal empathetic chatbot |
title_sort |
empathyear : an open-source avatar multimodal empathetic chatbot |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/9695 https://ink.library.smu.edu.sg/context/sis_research/article/10695/viewcontent/2024.acl_demos.7.pdf |
_version_ |
1819113105366450176 |