Machine learning based approach for sensory stimulated EEG signal classification

• Main Author: Tong, Chengxuan
• Other Authors: Guan Cuntai
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2025
• Subjects: Computer and Information Science; Deep learning; Brain-computer Interface
• Online Access: https://hdl.handle.net/10356/182925
• Institution: Nanyang Technological University

The human olfactory system, integral to perception, memory, and emotion, profoundly influences daily activities like eating and mood regulation. Studying odor responses is essential for advancements in food science and medical treatments. Currently, smell-related industries such as food, perfume, and aromatherapy heavily depend on professional odor panels to assess consumers’ preferences for scented products. However, recruiting such an odor panel is time consuming, costly and the panel might introduce subjective bias due to prior experience. EEG is ideal for decoding these olfactory processes due to its high temporal resolution, which captures the swift neural responses to scents, and its non-invasive nature, allowing for frequent, detailed observations of olfaction-related brain activity. A natural research question arises: How do we effectively decode olfactory processes from the Electroencephalogram (EEG)? Although some studies have attempted to decode EEG data using traditional machine learning methods, they often fail to effectively capture the complex temporal dynamics and spectral-spatial patterns inherent in EEG signals. Additionally, while breathing plays a crucial role in olfactory processes and could enhance decoding performance, suitable datasets for such studies are notably lacking. This thesis addresses these research gaps by developing advanced decoding algorithms and compiling a comprehensive multimodal olfactory dataset. In this thesis, we initially developed TESANet, a neural network that leverages spectral, spatial, and temporal dynamics for improved olfactory-EEG decoding. We further improved TESANet by proposing TASA, which minimizes spatial information loss and enhances temporal dynamics learning. Recognizing the critical role of breathing in olfaction, we compiled a comprehensive multimodal dataset. We proposed the Emotion Transformer to learn olfactory-induced emotional responses using our new dataset. Additionally, we utilized the breathing data in our collected dataset by proposing the TACAF framework, integrating EEG and breathing signals for superior classification performance. Our findings demonstrate significant improvements over traditional methods, highlighting the potential of computational approaches in olfactory analysis. A self-attention based neural network, TESANet is proposed for the initial study with olfactory-EEG. TESANet is designed to capture spatial-spectral features using a filter bank with spatial CNNs. We further utilize LSTM with self-attention to find the intercorrelation of the temporal segments for the temporal dynamics learning. The method was evaluated using a previously collected dataset from the Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore, consisting of 8 subjects. We achieved promising results with TESANet compared to both traditional machine learning approaches and other deep learning methods for other BCI tasks. We further improved TESANet by proposing a two-phase learning strategy with the Temporal Attention with Spatial Autoencoder Network (TASA). In this part of the thesis, we try to address two problems of the initial study. First, the information loss from spatial filtering by convolutional neural networks; second, the lack of temporal dynamics representation power from self-attention. We enhance spatial learning using a spatial autoencoder through reconstruction loss. Compared to conventional spatial convolutional neural networks, the first phase of training is conditioned on the reconstruction of the input using latent features, which allows us to obtain a spatial filter capable of performing spatial filtering with minimal spatial information loss. The spatial filter is then trained together with the downstream LSTM using a multi-head self-attention (MSA) module for temporal dynamics representation. We evaluate this method with the same dataset as the initial study. We demonstrate an improvement in performance and also perform substantial analysis on our results. As there is a lack of open-source olfactory-EEG data, and previously collected olfactory-EEG datasets consist of a limited number of subjects. Furthermore, there is a lack of studies reporting on olfactory-EEG analysis with breathing signals, as breathing is crucial for olfactory response. Thus, we collected a new olfactory-EEG dataset that consist of 20 subjects to address the above-mentioned issues. We collected the data using a scientific olfactometer for a well-controlled study. We then proposed a new method, Emotion Transformer (EmT) to capture multiple brain area connectivities as well as the short- and long-term contextual information that happened during the olfactory processes as the olfaction is closely related to emotions. Hence, the proposed EmT further captures spatial connectivity and temporal contextual information to improve decoding performance. EmT was evaluated with the collected olfactory EEG data and achieved promising results. To this end, none of the above methods take advantage of the additional breathing modality. We then utilized the dataset to build a multimodal deep learning framework that takes EEG and breathing signals as input. In this part of the thesis, we address the problem of automatic time window selection and also the incorporation of breathing data. We propose the Token Alignment and Cross-Attention Fusion network (TACAF), to classify odor stimuli by automatically selecting optimal time windows using multi-level wavelet-decomposition. We extract spatial features with a temporal module. We proposed a Temporal Token Semantic Alignment (TTSA) module to learn features from breathing data, ensuring synchronization with EEG for effective fusion The two modalities are merged with cross-attention fusion. Our results demonstrate that we achieved better performance than previous olfactory data analyses. Additionally, we compared several deep learning methods and showed that our multi-modal approach exhibited superior classification performance to existing methods. The improvements of the performance of olfactory-EEG decoding in this thesis demonstrated the potential of olfactory analysis with computational methods. Furthermore, the proposed algorithms are based on the neurophysiological features reported by previous studies.