Building generalizable deep learning solutions for mobile sensing

• Main Author: Xu, Huatao
• Other Authors: Mo Li
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Mobile sensing, IMU; IMU; Human activity recognition; Large language models; Penetrative AI
• Online Access: https://hdl.handle.net/10356/174809
• Institution: Nanyang Technological University

In an era where embedded and mobile devices are becoming ubiquitous, the intersection of Artificial Intelligence and the Internet of Things (AIoT) is rapidly transforming various fields. This thesis delves into the challenges and innovations in this domain, particularly focusing on the development of generalized sensing models for wearable sensor data. We introduce novel approaches to leverage the abundant unlabeled sensor data, physical sensing knowledge, and common knowledge embedded in Large Language Models (LLMs) to significantly enhance AIoT models. Firstly, we propose a foundational model for IoT sensor data processing, specifically focusing on Inertial Measurement Unit (IMU) data. This model, named LIMU-BERT, is inspired by self-supervised techniques in natural language processing and adapted for the multi-modal nature of sensor data. LIMU-BERT effectively learns generalizable representations from unlabeled sensor data, demonstrating superior performance in two typical sensing applications. Secondly, we introduce UniHAR, a universal learning framework for mobile devices. This framework employs physics-informed data augmentation techniques to address data heterogeneity in IMU-based human activity recognition. UniHAR, implemented on mobile platforms, showcases remarkable adaptability across various user groups and environments, outperforming existing solutions in cross-dataset model transfers. Lastly, the thesis explores the application of LLMs in AIoT, particularly in mobile sensing. We investigate the potential of LLMs, such as ChatGPT, to process IoT sensor data and accomplish real-world tasks. Our approach, termed "Penetrative AI", extends LLMs' capabilities beyond natural language processing and enables LLMs to interact with the physical world through IoT sensors/actuators, utilizing their embedded common-sense knowledge. The promising results from applications in user activity sensing and human heartbeat detection highlight the potential of LLMs to interpret sensor data and perform physical world tasks. The three works contribute significantly to the field of AIoT by developing novel methodologies that enhance the utility and adaptability of AIoT systems in diverse real-world scenarios.