M2Lens: Visualizing and explaining multimodal models for sentiment analysis

Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the...

Full description

Saved in:
Bibliographic Details
Main Authors: WANG, Xingbo, HE, Jianben, JIN, Zhihua, YANG, Muqiao, WANG, Yong, QU, Huamin
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2022
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/6777
https://ink.library.smu.edu.sg/context/sis_research/article/7780/viewcontent/21_TVCG_M2Lens.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-7780
record_format dspace
spelling sg-smu-ink.sis_research-77802022-02-24T06:55:52Z M2Lens: Visualizing and explaining multimodal models for sentiment analysis WANG, Xingbo HE, Jianben JIN, Zhihua YANG, Muqiao WANG, Yong QU, Huamin Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the complex intra- and inter-modal interactions between different communication channels. However, current multimodal models with strong performance are often deep-learning-based techniques and work like black boxes. It is not clear how models utilize multimodal information for sentiment predictions. Despite recent advances in techniques for enhancing the explainability of machine learning models, they often target unimodal scenarios (e.g., images, sentences), and little research has been done on explaining multimodal models. In this paper, we present an interactive visual analytics system, M2 Lens, to visualize and explain multimodal models for sentiment analysis. M2 Lens provides explanations on intra- and inter-modal interactions at the global, subset, and local levels. Specifically, it summarizes the influence of three typical interaction types (i.e., dominance, complement, and conflict) on the model predictions. Moreover, M2 Lens identifies frequent and influential multimodal features and supports the multi-faceted exploration of model behaviors from language, acoustic, and visual modalities. Through two case studies and expert interviews, we demonstrate our system can help users gain deep insights into the multimodal models for sentiment analysis. 2022-01-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/6777 info:doi/10.1109/TVCG.2021.3114794 https://ink.library.smu.edu.sg/context/sis_research/article/7780/viewcontent/21_TVCG_M2Lens.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Multimodal models sentiment analysis explainable machine learning Databases and Information Systems Numerical Analysis and Scientific Computing
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Multimodal models
sentiment analysis
explainable machine learning
Databases and Information Systems
Numerical Analysis and Scientific Computing
spellingShingle Multimodal models
sentiment analysis
explainable machine learning
Databases and Information Systems
Numerical Analysis and Scientific Computing
WANG, Xingbo
HE, Jianben
JIN, Zhihua
YANG, Muqiao
WANG, Yong
QU, Huamin
M2Lens: Visualizing and explaining multimodal models for sentiment analysis
description Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the complex intra- and inter-modal interactions between different communication channels. However, current multimodal models with strong performance are often deep-learning-based techniques and work like black boxes. It is not clear how models utilize multimodal information for sentiment predictions. Despite recent advances in techniques for enhancing the explainability of machine learning models, they often target unimodal scenarios (e.g., images, sentences), and little research has been done on explaining multimodal models. In this paper, we present an interactive visual analytics system, M2 Lens, to visualize and explain multimodal models for sentiment analysis. M2 Lens provides explanations on intra- and inter-modal interactions at the global, subset, and local levels. Specifically, it summarizes the influence of three typical interaction types (i.e., dominance, complement, and conflict) on the model predictions. Moreover, M2 Lens identifies frequent and influential multimodal features and supports the multi-faceted exploration of model behaviors from language, acoustic, and visual modalities. Through two case studies and expert interviews, we demonstrate our system can help users gain deep insights into the multimodal models for sentiment analysis.
format text
author WANG, Xingbo
HE, Jianben
JIN, Zhihua
YANG, Muqiao
WANG, Yong
QU, Huamin
author_facet WANG, Xingbo
HE, Jianben
JIN, Zhihua
YANG, Muqiao
WANG, Yong
QU, Huamin
author_sort WANG, Xingbo
title M2Lens: Visualizing and explaining multimodal models for sentiment analysis
title_short M2Lens: Visualizing and explaining multimodal models for sentiment analysis
title_full M2Lens: Visualizing and explaining multimodal models for sentiment analysis
title_fullStr M2Lens: Visualizing and explaining multimodal models for sentiment analysis
title_full_unstemmed M2Lens: Visualizing and explaining multimodal models for sentiment analysis
title_sort m2lens: visualizing and explaining multimodal models for sentiment analysis
publisher Institutional Knowledge at Singapore Management University
publishDate 2022
url https://ink.library.smu.edu.sg/sis_research/6777
https://ink.library.smu.edu.sg/context/sis_research/article/7780/viewcontent/21_TVCG_M2Lens.pdf
_version_ 1770576067040378880