Deep learning for fusing speech and text for detection of Alzheimer’s Disease

• Main Author: Tan, Jun Xian
• Other Authors: Jagath C Rajapakse
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/156350
• Institution: Nanyang Technological University

Alzheimer’s Disease (AD) is one of the most common forms of dementia which occurs mainly in elderlies. Extensive research has been done to find a cure. However, early intervention is just as important. Impaired speech, which occurs in the early stages of AD, could be used as a biomarker for early detection of AD. Impaired speech could provide useful information, such as audio and text features, to be used for detection of AD. In this project, we will be using speech and text separately to detect AD. Then, speech and text will be combined to detect AD. Audio features like extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) and text features like document embeddings, word embeddings, will be extracted. These features will then be used for detection of AD, using different machine learning and deep learning methods, such as Decision Tree, Random Forest, Support Vector Machine, Logistic Regression, Neural Networks. To obtain better results, the best models achieved from using speech and text will be combined. The fusion of speech and text will be done by using fusion mechanisms such as Concatenation and Bilinear Pooling. Our results indicated that by using the bilinear pooling mechanism, it produced better results compared to the concatenation mechanism.