Sensing enhanced mobile computing with smart devices

• Main Author: Jiang, Shiqi
• Other Authors: Li Mo
• Format: Theses and Dissertations
• Language: English
• Published: 2017
• Subjects: DRNTU::Engineering::Computer science and engineering
• Online Access: http://hdl.handle.net/10356/73055
• Institution: Nanyang Technological University

Today's mobile devices not only serve as the key communication tools, but become important sensing and computing platforms as well. Sensing capacities promise the emergence of sensing enhanced mobile computing on smart devices, which leverages various on-board sensors, applies sophisticated machine learning techniques for intelligent reasoning according to the context inference of surrounding environments. In this report, we introduce two systems we built based on this idea, which are PDS, a Phantom Data Usage Detection System and Memento, an Emotion Driven Lifelogging System. First we introduce PDS, a phantom data usage detection system. With the wide development of smartphones, mobile data usage has enjoyed rapid growth in recent years. Unfortunately many users are plagued with Phantom Data Usage (PDU), which refers to the unexpected mobile data usage that does not accord with user perception. We investigate real PDU issues and find the causes of PDU are not only the exceptions of applications, e.g., software bugs or malware, but also the user's personalized misuse. Based on the observations that each user preserves specific data usage patterns under particular environmental context, we present PDS, which automatically detects whether the current data usage is consumed as expected. Then we present Memento, an emotion driven lifelogging system on wearables. Due to the increasing popularity of mobile devices, the usage of lifelogging has been dramatically expanded. People collect their daily memorial moments and share with friends on the social network, which has been an emerging lifestyle. We see great potential of lifelogging applications along with rapid growth of recent wearable market, where more sensors are introduced to wearables, i.e., electroencephalogram (EEG) sensors, that can further sense the user's mental activities, e.g., emotions. We present the design and implementation of Memento, an emotion driven lifelogging system on wearables. Memento integrates EEG sensors with smart glasses. Since memorable moments usually coincides with the user's emotional changes, Memento leverages the knowledge from the brain-computer-interface (BCI) domain to analyze the EEG signals to infer emotions and automatically launch lifelogging based on that. Towards building Memento on COTS wearable devices, we study EEG signals in mobility cases and propose a multiple sensor fusion based approach to estimate signal quality. We present a customized two-phase emotion recognition architecture, considering both the affordability and efficiency of wearable-class devices. We also discuss the optimization framework to automatically choose and configure the suitable lifelogging method (video, audio or image) by analyzing the environment and system context. Finally our experimental evaluation shows that Memento is responsive, efficient and user-friendly on wearables.