Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition

Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous predict...

Full description

Saved in:
Bibliographic Details
Main Authors: Zhang, Su, Tang, Chuangao, Guan, Cuntai
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2022
Subjects:
Online Access:https://hdl.handle.net/10356/161791
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-161791
record_format dspace
spelling sg-ntu-dr.10356-1617912022-09-20T05:16:12Z Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition Zhang, Su Tang, Chuangao Guan, Cuntai School of Computer Science and Engineering Engineering::Computer science and engineering Continuous Emotion Recognition Knowledge Distillation Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous prediction of the EEG modality by using the dark knowledge from the visual modality. The teacher model is built by a cascade convolutional neural network - temporal convolutional network (CNN-TCN) architecture, and the student model is built by TCNs. They are fed by video frames and EEG average band power features, respectively. Two data partitioning schemes are employed, i.e., the trial-level random shuffling (TRS) and the leave-one-subject-out (LOSO). The standalone teacher and student can produce continuous prediction superior to the baseline method, and the employment of the visual-to-EEG cross-modal KD further improves the prediction with statistical significance, i.e., p-value <0.01 for TRS and p-value <0.05 for LOSO partitioning. The saliency maps of the trained student model show that the brain areas associated with the active valence state are not located in precise brain areas. Instead, it results from synchronized activity among various brain areas. And the fast beta and gamma waves, with the frequency of 18−30Hz and 30−45Hz, contribute the most to the human emotion process compared to other bands. The code is available at https://github.com/sucv/Visual_to_EEG_Cross_Modal_KD_for_CER. Agency for Science, Technology and Research (A*STAR) Published version This work was supported by the RIE2020 AME Programmatic Fund, Singapore (No. A20G8b0102). 2022-09-20T05:16:12Z 2022-09-20T05:16:12Z 2022 Journal Article Zhang, S., Tang, C. & Guan, C. (2022). Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition. Pattern Recognition, 130, 108833-. https://dx.doi.org/10.1016/j.patcog.2022.108833 0031-3203 https://hdl.handle.net/10356/161791 10.1016/j.patcog.2022.108833 2-s2.0-85131463243 130 108833 en A20G8b0102 Pattern Recognition © 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Continuous Emotion Recognition
Knowledge Distillation
spellingShingle Engineering::Computer science and engineering
Continuous Emotion Recognition
Knowledge Distillation
Zhang, Su
Tang, Chuangao
Guan, Cuntai
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
description Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous prediction of the EEG modality by using the dark knowledge from the visual modality. The teacher model is built by a cascade convolutional neural network - temporal convolutional network (CNN-TCN) architecture, and the student model is built by TCNs. They are fed by video frames and EEG average band power features, respectively. Two data partitioning schemes are employed, i.e., the trial-level random shuffling (TRS) and the leave-one-subject-out (LOSO). The standalone teacher and student can produce continuous prediction superior to the baseline method, and the employment of the visual-to-EEG cross-modal KD further improves the prediction with statistical significance, i.e., p-value <0.01 for TRS and p-value <0.05 for LOSO partitioning. The saliency maps of the trained student model show that the brain areas associated with the active valence state are not located in precise brain areas. Instead, it results from synchronized activity among various brain areas. And the fast beta and gamma waves, with the frequency of 18−30Hz and 30−45Hz, contribute the most to the human emotion process compared to other bands. The code is available at https://github.com/sucv/Visual_to_EEG_Cross_Modal_KD_for_CER.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Zhang, Su
Tang, Chuangao
Guan, Cuntai
format Article
author Zhang, Su
Tang, Chuangao
Guan, Cuntai
author_sort Zhang, Su
title Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
title_short Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
title_full Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
title_fullStr Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
title_full_unstemmed Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
title_sort visual-to-eeg cross-modal knowledge distillation for continuous emotion recognition
publishDate 2022
url https://hdl.handle.net/10356/161791
_version_ 1745574632390393856