Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition
Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous predict...
Saved in:
Main Authors: | , , |
---|---|
Other Authors: | |
Format: | Article |
Language: | English |
Published: |
2022
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/161791 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
id |
sg-ntu-dr.10356-161791 |
---|---|
record_format |
dspace |
spelling |
sg-ntu-dr.10356-1617912022-09-20T05:16:12Z Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition Zhang, Su Tang, Chuangao Guan, Cuntai School of Computer Science and Engineering Engineering::Computer science and engineering Continuous Emotion Recognition Knowledge Distillation Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous prediction of the EEG modality by using the dark knowledge from the visual modality. The teacher model is built by a cascade convolutional neural network - temporal convolutional network (CNN-TCN) architecture, and the student model is built by TCNs. They are fed by video frames and EEG average band power features, respectively. Two data partitioning schemes are employed, i.e., the trial-level random shuffling (TRS) and the leave-one-subject-out (LOSO). The standalone teacher and student can produce continuous prediction superior to the baseline method, and the employment of the visual-to-EEG cross-modal KD further improves the prediction with statistical significance, i.e., p-value <0.01 for TRS and p-value <0.05 for LOSO partitioning. The saliency maps of the trained student model show that the brain areas associated with the active valence state are not located in precise brain areas. Instead, it results from synchronized activity among various brain areas. And the fast beta and gamma waves, with the frequency of 18−30Hz and 30−45Hz, contribute the most to the human emotion process compared to other bands. The code is available at https://github.com/sucv/Visual_to_EEG_Cross_Modal_KD_for_CER. Agency for Science, Technology and Research (A*STAR) Published version This work was supported by the RIE2020 AME Programmatic Fund, Singapore (No. A20G8b0102). 2022-09-20T05:16:12Z 2022-09-20T05:16:12Z 2022 Journal Article Zhang, S., Tang, C. & Guan, C. (2022). Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition. Pattern Recognition, 130, 108833-. https://dx.doi.org/10.1016/j.patcog.2022.108833 0031-3203 https://hdl.handle.net/10356/161791 10.1016/j.patcog.2022.108833 2-s2.0-85131463243 130 108833 en A20G8b0102 Pattern Recognition © 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). application/pdf |
institution |
Nanyang Technological University |
building |
NTU Library |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
NTU Library |
collection |
DR-NTU |
language |
English |
topic |
Engineering::Computer science and engineering Continuous Emotion Recognition Knowledge Distillation |
spellingShingle |
Engineering::Computer science and engineering Continuous Emotion Recognition Knowledge Distillation Zhang, Su Tang, Chuangao Guan, Cuntai Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
description |
Visual modality is one of the most dominant modalities for current continuous emotion recognition methods. Compared to which the EEG modality is relatively less sound due to its intrinsic limitation such as subject bias and low spatial resolution. This work attempts to improve the continuous prediction of the EEG modality by using the dark knowledge from the visual modality. The teacher model is built by a cascade convolutional neural network - temporal convolutional network (CNN-TCN) architecture, and the student model is built by TCNs. They are fed by video frames and EEG average band power features, respectively. Two data partitioning schemes are employed, i.e., the trial-level random shuffling (TRS) and the leave-one-subject-out (LOSO). The standalone teacher and student can produce continuous prediction superior to the baseline method, and the employment of the visual-to-EEG cross-modal KD further improves the prediction with statistical significance, i.e., p-value <0.01 for TRS and p-value <0.05 for LOSO partitioning. The saliency maps of the trained student model show that the brain areas associated with the active valence state are not located in precise brain areas. Instead, it results from synchronized activity among various brain areas. And the fast beta and gamma waves, with the frequency of 18−30Hz and 30−45Hz, contribute the most to the human emotion process compared to other bands. The code is available at https://github.com/sucv/Visual_to_EEG_Cross_Modal_KD_for_CER. |
author2 |
School of Computer Science and Engineering |
author_facet |
School of Computer Science and Engineering Zhang, Su Tang, Chuangao Guan, Cuntai |
format |
Article |
author |
Zhang, Su Tang, Chuangao Guan, Cuntai |
author_sort |
Zhang, Su |
title |
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
title_short |
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
title_full |
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
title_fullStr |
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
title_full_unstemmed |
Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognition |
title_sort |
visual-to-eeg cross-modal knowledge distillation for continuous emotion recognition |
publishDate |
2022 |
url |
https://hdl.handle.net/10356/161791 |
_version_ |
1745574632390393856 |