Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Objective. The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two pre...

Full description

Saved in:
Bibliographic Details
Main Authors: Zhang, Yuzhe, Liu, Huan, Wang, Di, Zhang, Dalin, Lou, Tianyu, Zheng, Qinghua, Quek, Chai
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2024
Subjects:
Online Access:https://hdl.handle.net/10356/179031
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-179031
record_format dspace
spelling sg-ntu-dr.10356-1790312024-07-16T07:43:54Z Cross-modal credibility modelling for EEG-based multimodal emotion recognition Zhang, Yuzhe Liu, Huan Wang, Di Zhang, Dalin Lou, Tianyu Zheng, Qinghua Quek, Chai School of Computer Science and Engineering Computer and Information Science Multimodal emotion recognition Sequential pattern consistency Objective. The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals. Approach. In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility. Main results. We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments. Significance. All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition. This work was supported by National Natural Science Foundation of China (62192781, 62202367), Project of China Knowledge Centre for Engineering Science and Technology, Project of Chinese academy of engineering ‘The Online and Offline Mixed Educational Service System for ‘The Belt and Road’ Training in MOOC China’. 2024-07-16T07:43:17Z 2024-07-16T07:43:17Z 2024 Journal Article Zhang, Y., Liu, H., Wang, D., Zhang, D., Lou, T., Zheng, Q. & Quek, C. (2024). Cross-modal credibility modelling for EEG-based multimodal emotion recognition. Journal of Neural Engineering, 21(2), 026040-. https://dx.doi.org/10.1088/1741-2552/ad3987 1741-2560 https://hdl.handle.net/10356/179031 10.1088/1741-2552/ad3987 38565099 2-s2.0-85190332957 2 21 026040 en Journal of Neural Engineering © 2024 IOP Publishing Ltd. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Computer and Information Science
Multimodal emotion recognition
Sequential pattern consistency
spellingShingle Computer and Information Science
Multimodal emotion recognition
Sequential pattern consistency
Zhang, Yuzhe
Liu, Huan
Wang, Di
Zhang, Dalin
Lou, Tianyu
Zheng, Qinghua
Quek, Chai
Cross-modal credibility modelling for EEG-based multimodal emotion recognition
description Objective. The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals. Approach. In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility. Main results. We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments. Significance. All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Zhang, Yuzhe
Liu, Huan
Wang, Di
Zhang, Dalin
Lou, Tianyu
Zheng, Qinghua
Quek, Chai
format Article
author Zhang, Yuzhe
Liu, Huan
Wang, Di
Zhang, Dalin
Lou, Tianyu
Zheng, Qinghua
Quek, Chai
author_sort Zhang, Yuzhe
title Cross-modal credibility modelling for EEG-based multimodal emotion recognition
title_short Cross-modal credibility modelling for EEG-based multimodal emotion recognition
title_full Cross-modal credibility modelling for EEG-based multimodal emotion recognition
title_fullStr Cross-modal credibility modelling for EEG-based multimodal emotion recognition
title_full_unstemmed Cross-modal credibility modelling for EEG-based multimodal emotion recognition
title_sort cross-modal credibility modelling for eeg-based multimodal emotion recognition
publishDate 2024
url https://hdl.handle.net/10356/179031
_version_ 1806059758027800576