Deep CNN-LSTM supervised model and CNN self-supervised model for human activity recognition

• Main Author: Liao, Zixin
• Other Authors: Kwoh Chee Keong
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/166096
• Institution: Nanyang Technological University

Human Activity Recognition (HAR) has garnered significant interest from researchers in past decades. With the quick development of wearable sensor technology and the high availability of smart devices, e.g., accelerometers and gyroscopes embedded in smartphones, HAR has become a popular field of research recently. In this paper, we propose a framework for HAR data classification which automatically extracts spatial and temporal features from smart device sensory data. This is achieved via a hybrid supervised learning architecture, that consists of a Convolutional Neural Network (CNN), and a Long Short-Term Memory Network (LSTM). However, a large amount of labeled data is typically required to perform supervised learning, which can be lacking due to data privacy concerns and the high cost of manual labeling in real-world scenarios. Therefore, learning from the large amounts of unlabeled data becomes crucial. To this end, we propose a self-supervised learning (SSL) framework that learns useful representations from unlabeled HAR sensory data. Our framework consists of two stages: 1) self-supervised pretraining, where we propose a set of pretext tasks to help the model learn from unlabeled data, and 2) fine-tuning the pre-trained model with the few available labeled samples according to the original HAR task. The results demonstrate that our SSL approach significantly improves the model performance compared to supervised training given limited labeled samples. In addition, by fine-tuning the simple 1-D CNN pre-trained self-supervised model using only 5% of labeled data, we can attain a level of performance that is comparable to the complex CNN-LSTM supervised training with full labels. Last, we observe that self-supervised pre-training assists the models in developing robustness to data imbalanced issues. The source code is available on https: //github.com/LizLicense/HAR-CNN-LSTM-ATT-pyTorch.git.