Enabling real time in-situ context based experimentation to observe user behaviour

• Main Author: MURALIDARAN, Kartik
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2015
• Subjects: context aware computing; mobile computing; event processing; context uncertainty; mobile application; natural language processing; behavioural experimentation.; Numerical Analysis and Scientific Computing; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/etd_coll/128; https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=1122&context=etd_coll
• Institution: Singapore Management University

Today’s mobile phones represent a rich and powerful computing platform, given their sensing, processing and communication capabilities. These devices are also part of the everyday life of millions of people, and coupled with the unprecedented access to personal context, make them the ideal tool for conducting behavioural experiments in an unobtrusive way. Transforming the mobile device from a mere observer of human context to an enabler of behavioural experiments however, requires not only providing experimenters access to the deep, near-real time human context (e.g., location, activity, group dynamics) but also exposing a disciplined scientific experimentation service that frees them from the many experimental chores such as subject selection and mitigating biases. This dissertation shows that it is possible to enable insitu real-time experimentation that require context-specific triggers targeting real participants on their actual mobile phones. I first developed a platform called Jarvis that allows experimenters to easily, and quickly create a diverse range of observational and treatment studies, specify a variety of opportune moments for targeting participants, and support multiple intervention (treatment) content types. Jarvis automates the process of participant selection and the creation of experimental groups, and adheres to the well known randomized controlled trial (RCT) experimental process. Of the many possibilities, a use case I envision for Jarvis, is providing retailers a platform to run lifestyle based experiments that investigate promotional strategies. Such experiments might entail the platform to provide the experimenter with the appropriate target population based on their preferences. To support this, I developed a matching and scoring algorithm that accurately factors participants’ preferences when matching experiment promotions and is capable of combining structured and unstructured promotion information into a single score. Doing so, will allow the experimentation system to target the right set of participants. Finally, I developed techniques for capturing and handling context uncertainty within Jarvis. As the opportune experiment-intervention moments are identified from sources such as sensors and social media, which have inherent uncertainties associated with them, it is crucial that such information is recorded and/or processed. More specifically, Jarvis defines a confidence metric for the location predicate as well as dynamically computes the sample size for a given experiment under context uncertainty. In doing so it provides adequate information to the experimenter to process the results of an experiment in addition to maximizing the statistical power. I validated my dissertation in the following way. Through a series of live experiments I showcase the diversity of the system in supporting multiple experiment designs, the ease of experiment specification, and the rich behavioural information accessible to the experimenter in the form of a report. The matching and scoring algorithm was evaluated in two different ways; First, an in-depth analytical evaluation of the ranking algorithm was conducted to understand the accuracy of the algorithm. Second, I ran a user study with 43 undergraduate students to understand the effectiveness of the algorithm. Finally, I validate the context-uncertainty handling capabilities of Jarvis through simulations and show that using overlap ratios to represent location confidence is reliable and that the algorithm to estimate the number of false positives has minimal errors. Both these values are important in understanding the outcome of an experiment and in turn defining it’s success criteria.