Understanding and improving interactive systems design with human-in-the-loop machine learning

• Main Author: Zhang, Yaqian
• Other Authors: Goh Wooi Boon
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2020
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence; Engineering::Computer science and engineering::Computer applications::Social and behavioral sciences
• Online Access: https://hdl.handle.net/10356/136763
• Institution: Nanyang Technological University

New developments in machine learning techniques have created opportunities for the Human-Computer Interaction (HCI) community to incorporate more intelligent means to improve and enhance user experience during the interaction. This thesis starts by exploring and identifying a suitable role where machine learning algorithms can play to improve the design of interactive systems. Once the area has been identified, a suitable learning algorithm has been designed and evaluated to ensure it is able to address the constraints posed by human-in-the-loop interactive systems. Firstly, a user study was conducted to explore design factors that might influence users’ performance during competitive and cooperative gameplay. A key observation was that there is a significant performance decline when the disparity in abilities between the gaming partners is large. This result suggests that to maintain a high level of cognitive engagement, performance disparity among group members needs to be moderated. In order to reduce this disparity, stronger users should be challenged with harder tasks and less competent users should be presented with easier tasks. In short, automatic task difficulty adaption was seen to be an important area in improving the design of interactive systems and maintaining user performance. This motivates the subsequent research on how the difficulty level of a series of tasks can be autonomously adjusted based on each individual's ability, especially in relation to the design of responsive intelligent tutoring systems. Traditionally, difficulty level is often determined by domain experts based on some hand-crafted rules. However, with the adoption of Massive Open Online Courses (MOOCs), it has become harder to manually personalize task difficulty as the system designers are faced with a very large question bank and a user base consisting of individuals with diverse backgrounds and ability levels. This research focuses on developing a data-driven method to adaptively adjust difficulty levels in order to maintain a target user performance in a visual memory task the difficulty level of which is highly variable among different individuals. The first challenge is to obtain personalized difficulty ranking. This was addressed using a clustering-based method which can learn a personalized difficulty ranking based on pre-collected data. A novel general method for determining the number of clusters was proposed by exploiting the curvature information in the clustering objective function. Unlike existing methods that often require substantiated computational resources and parametric assumptions, the proposed approach is computationally efficient and suitable for use in real-time interactive applications. The next challenge is the issue of difficulty adjustment which was formulated as a reinforcement learning problem. Reinforcement learning (RL) is a class of machine learning algorithms which is concerned with sequential decision making. Unlike traditional RL problems, like controlling robots (MuJoCo) or playing board games (game of Go), where accurate simulators exist, the environment being considered here is a typical HCI system that involves a human in the loop. The cost of taking a sample is thus a critical consideration as it directly affects user interaction experience and impacts the perceived responsiveness and “intelligence” of the interactive system. In addition, unlike many recent RL studies, the action space considered in this work is significantly larger, comprising of hundreds of different visual memory tasks. To address these constraints, a novel bootstrapped policy gradient (BPG) framework was developed, which can incorporate prior knowledge of difficulty ranking into policy gradient to enhance sample efficiency. BPG was applied to solve the difficulty adaptation problem in the challenging RL environment comprising of large action spaces and short horizon, and was demonstrated to be able to achieve fast and unbiased convergence both in theory and in practice.