Neurophysiology-inspired neural networks for affective brain-computer interfaces

• Main Author: Ding, Yi
• Other Authors: Guan Cuntai
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/170172
• Institution: Nanyang Technological University

Brain-computer interface (BCI) facilitates the computer's perception of human emotional mental states through electroencephalogram (EEG), also known as affective brain-computer interface (aBCI). An aBCI can be applied in emotion-focused therapy (EFT) for psychological disabilities, such as generalized anxiety disorder (GAD) and autistic spectrum disorders (ASD). To ensure reliable real-world implementation, an effective aBCI system requires accurate emotion detection and high generalization capability of decoding algorithms. However, emotion recognition using EEG remains challenging due to factors such as low signal-to-noise ratio (SNR), non-stationary nature of EEG signals, and the complexity of cognitive processes underlying emotions. In recent times, deep learning methods have demonstrated promising results across various domains, including computer vision, natural language processing, audio signal processing, and several sub-areas of BCI. Successful implementation of deep learning methods relies on factors such as abundant data, effective training methods, and access to ample computational resources. Equally important is the design of a well-structured neural network architecture capable of capturing the intrinsic characteristics of the data. Through a comprehensive evaluation of state-of-the-art (SOTA) methods in the BCI domain under various assessment settings, this thesis highlights the persistent challenges faced in EEG-based emotion recognition tasks, particularly in generalized evaluation scenarios. This thesis focuses on incorporating insights from neurophysiology studies into neural network design, aiming to enhance the decoding performance of emotions using EEG signals. In this thesis, a novel multi-scale convolutional neural network called TSception is proposed as the primary contribution. TSception aims to capture both temporal dynamics and spatial asymmetry in EEG signals for effective emotion recognition. Neurophysiology studies suggest that emotional information is associated with brain activities across multiple frequency bands, and asymmetrical spatial patterns are observed during emotional cognitive processes. Taking this prior knowledge into account, the neural network design incorporates a multi-scale temporal convolutional layer and an asymmetric spatial convolutional layer. These layers are specifically designed to capture dynamic temporal/frequency information and spatial asymmetry patterns present in EEG signals. To evaluate the performance of TSception, extensive experiments are conducted on two benchmarking datasets. The results of these experiments are compared with several shallow and deep learning methods. TSception consistently achieves higher classification accuracies and F1 scores compared to other methods across most of the experiments. These findings underscore the effectiveness of TSception in EEG-based emotion recognition tasks. This thesis introduces a novel graph neural network, referred to as LGGNet, which is proposed for emotion recognition. LGGNet incorporates the understanding from neuropsychological knowledge that cooperative activities among different brain functional areas contribute to the experience of emotions. To capture temporal dynamic information, a multi-scale 1D convolutional layer is employed. In contrast to TSception, LGGNet utilizes a power activation layer to enable the network to learn power-related features from EEG signals. Additionally, a kernel-level attentive fusion mechanism is devised to combine the learned temporal information effectively. Incorporating neurophysiologically meaningful local and global graphs, LGGNet includes local and global graph filtering layers that enable the modeling of complex relationships within and among functional areas of the brain. Notably, LGGNet exhibits improvements over TSception in emotion recognition tasks. Moreover, due to the network architecture's meaningful design based on neurophysiological principles, LGGNet achieves superior classification results compared to several SOTA methods in tasks involving attention, fatigue, and preference classification. By enabling the temporal convolutional neural networks to learn intricate spatial information from EEG signals, this thesis improves the decoding performances for both EEG emotion classification and regression tasks. The proposed model, named MASA-TCN which stands for \textbf{M}ulti-\textbf{A}nchor \textbf{S}pace-\textbf{A}ware \textbf{T}emporal \textbf{C}onvolutional neural \textbf{N}etworks, incorporates a space-aware temporal layer that enables TCNs to additionally learn from spatial relations among EEG electrodes. Moreover, MASA-TCN can capture dynamic temporal dependencies through the utilization of a novel multi-anchor block with attentive fusion. Extensive experiments on two publicly available datasets show MASA-TCN achieves higher results than the SOTA methods for both EEG emotion regression and classification tasks. The improvements in the decoding performances achieved by the proposed algorithms indicate the effectiveness of neurophysiologically meaningful neural network design. It is noteworthy that all the evaluation settings employed in this thesis adhere to generalized settings, where the model is never exposed to the test data during training. This thesis also publishes the source codes of the proposed methods to benefit the research community.